Materialized commits don't report errors nicely. about pachyderm

[neojski]

Apparently, even the error isn't logged. I wanted to play with pachyderm so I ran

 {
   "type": "map",
   "input": "identity",
   "image": "ubuntu",
   "command" : ["/bin/echo"]
 }

Hoping to see some output. But log only shows me:

mapreduce.go:478: Running job:
identity
mapreduce.go:493: Job: {map identity ubuntu [/bin/echo] 0 0 0}
mapreduce.go:46: spinupContainer ubuntu [/bin/echo]

and nothing more. There's no result anywhere and no error.

[jdoliner]

Interesting, there should at least be some logging for the files that get posted in to the server. Is there anything in the identity directory in pfs?

Either way this case should definitely give users more of an idea what's going on than it currently does.

[jdoliner]

A few comments on this issue in general:

I removed the v0.4 label because this obviously won't be shipping in that release. We've had to to rethink this issue a bit because materializing now happens asynchronously. This is good because materializing can take a while which may lead to timeouts and thus reported error message will be lost.

What if we added a route like:

pfs/job/foo/log

which would return stderr from the job.

Edit: This would also work nicely because you could see the results of materializing on a particular commit using pfs/jobs/foo/log?commit=<commit>

[neojski]

I think it'd be useful to have some blocking call that would stream the logs to my local machine. I'd like to know when a job is finished. I would like to have timestamps for logged messages.

How is running the job on multiple machines implemented? I think, depending on that the logging could be also complicated.

[jdoliner]

I think it'd be useful to have some blocking call that would stream the logs to my local machine. I'd like to know when a job is finished. I would like to have timestamps for logged messages.

Definitely agree on the value of both of these things. My idea for how this would work is that calls to get logs pfs/job/foo/log and job output pfs/job/foo/file/... would block until the job completed. So for example if I do:

curl pfs/job/foo/log?commit=<commit>

then that request will block until the job foo is materialize for <commit>

How is running the job on multiple machines implemented? I think, depending on that the logging could be also complicated.

Each shard runs the job for its local data, recording the output from stderr of all of the containers it runs. This definitely makes collecting the logs a bit trickier. I'm not sure what it means for the interface for getting logs. Maybe we'd want a way to get logs from a particular physical machine or shard?

[JoeyZwicker]

New pps supports errors nicely.