What can cause differences in speed of performing certain actions? about bazel-buildfarm

[SinOverCos]

Submitted the issue without the description by accident, edited with my actual question ^

[SinOverCos]

Posted an example configuration that demonstrates this problem when building tools_remote: #1343 (comment)

[werkt]

This action in particular, ImageLayer to produce a tar, is a proportionally labor intensive task: a small perturbation in the input set always creates a differentiated output that is proportionate to the whole input set size. This is also an IO, rather than compute, bound task, which will be exaggerated by the iops capacity of your buildfarm workers, leaving you in the cpu/ram unused state you see.

I recommend using --remote_grpc_log to enumerate the requests being performed and build a timescale of each request to break down where the performance gap exists. These will overlap, but there will be some obvious criticality to the request sequence: FMB will complete before Writes, which will complete before Execute, which will complete before Reads. Network and disk IO measurements during the execution period will also be illuminating, if either of them happen to be pegged to their capacities.

Such a breakdown will point by comparison between systems to the capacity for improvements, and let you know where your bottleneck exists, and where engineering effort for buildfarm could improve them.

[SinOverCos]

Thanks, @werkt . I suspected that something about the IO would be the bottleneck here and will use your advice in tracking down the exact bottleneck. In general, would you recommend that docker image rules be executed locally or are there cases that still merit executing remotely?

Separately, I have an example of (what I think is) a non-IO-bound build that takes much longer on remote than on local -- would you be able to take a look at that? Much appreciated! #1343

[werkt]

I strongly recommend executing that particular item locally. There has been historically some issues identifying these with tags (not sure precisely the current state), but the failsafe way is the one Alex Eagle recommends here: https://blog.aspect.dev/bazelrc-flags (look for --modify_execution_info). If you substitute PackageTar for ImageLayer (and definitely the digest summing mnemonics of rules_docker), you will be able to ensure they run locally; Even a 9x cost of the 'commercial' RE is too much to pay, and you've got to be concerned in that instance in how much bandwidth you're attributing to each individual request out of the budget for a machine.

Will have a look at that task regarding non-IO compromises