Worker linkInputs stuck in fatal state about bazel-buildfarm

[werkt]

It looks like a referenced CAS entry was expired - I would try bumping up your CAS byte count in your server's configuration as a test.

As to why this requires a server shutdown, I'm unsure: can you identify the rev of buildfarm running on both worker and server?

I have some pending changes to make the worker survive something like this a little more gracefully - in that case the work will be requeued for server validation. I will prioritize getting those changes into bazelbuild/master regardless.

[pcj]

As to why this requires a server shutdown, I'm unsure: can you identify the rev of buildfarm running on both worker and server?

(these are images built off #128)

[pcj]

I've set the CAS max size to LONG.MAX_VALUE, so I don't think that's it.

The worker ends up requesting a zero-length blob from the server that is NOT_FOUND. That blob hash does not seem to be populated in the contentaddressablestorage, so I think the reply from the server is correct. Still looking into it.

read request resource_name: "default_memory_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0"

readBlob default_memory_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0 using instance build.buildfarm.instance.memory.MemoryInstance@7fa1cd21
getBlob(hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
, 0, 0) --> NULL build.buildfarm.instance.memory.MemoryLRUContentAddressableStorage@53941ba3

[werkt]

ah hah! you need a recent change that prevents empty blob fetches. Note the /0 at the end of that digest name

[pcj]

Was that merged already? Again I'm currently off #128 but can cherry-pick that.

[pcj]

I think that's a good change, but I applied it directly and has not solved the problem (does not suppress that request to the server). Here's the file it's trying to link:

linking /tmp/worker/default_memory_instance/operations/d2b5da3b-dc85-4283-983b-0ae14aefd31e/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/ROOT -> hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

That file actually exists (non-remote build) and is empty:

du -h bazel-bin/external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/ROOT 
0	bazel-bin/external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/ROOT

Perhaps a change in CASFileCache.put to skip the request for zero-length blob? (just create a symlink right away, skip the tmpPath step)

I have to shift gears here soon but will continue later in the day.

[werkt]

similar to #150, can you provide a remote.log of the action in question? it will make incremental debugging of this issue easier.