Remote builds stuck on Buildfarm that is deployed with Helm. about bazel-buildfarm

[monaka]

I tried a remote cache (CAS) in the same environment also.
I suppose CAS works well.

[monaka]

This issue might be somewhat related to #1749

[werkt]

I just tried the helm install with minikube, with our recommended port forwarding specification, had no trouble building your example program.

Using bazel-buildfarm-server.bazel-buildfarm.svc as a name to contact the running service implies that you have some association between the cluster and the client bazel environment - are you sure that you can contact it - see bf-cat - at all for Capabilities (the most basic communication) on the port indicated? After verifying that bazel-buildfarm-server.bazel-buildfarm.svc resolves, of course.

[monaka]

bazel-buildfarm-server.bazel-buildfarm.svc is resolved.

% kubectl get svc -n bazel-buildfarm
NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
bazel-buildfarm-redis-headless   ClusterIP   None             <none>        6379/TCP            112d
bazel-buildfarm-redis-master     ClusterIP   10.100.239.216   <none>        6379/TCP            112d
bazel-buildfarm-redis-replicas   ClusterIP   10.100.51.9      <none>        6379/TCP            112d
bazel-buildfarm-server           ClusterIP   10.100.76.80     <none>        8980/TCP,9090/TCP   112d
bazel-buildfarm-shard-worker     ClusterIP   10.100.6.122     <none>        8982/TCP,9090/TCP   112d

% kubectl exec -it aaos-app-7f8479b66d-qwr9b -- nslookup bazel-buildfarm-server.bazel-buildfarm.svc
Server:         10.100.0.10
Address:        10.100.0.10#53

Name:   bazel-buildfarm-server.bazel-buildfarm.svc.cluster.local
Address: 10.100.76.80

And as I wrote above, it seems that CAS works fine. Just not working remote_execition only.

$ bazel clean
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ bazel run --remote_cache=grpc://bazel-buildfarm-server.bazel-buildfarm.svc:8980 :main
INFO: Invocation ID: 4e7c1beb-3f46-4032-b6bb-f0061b2e42bd
INFO: Analyzed target //:main (83 packages loaded, 382 targets configured).
INFO: Found 1 target...
Target //:main up-to-date:
  bazel-bin/main
INFO: Elapsed time: 0.477s, Critical Path: 0.06s
INFO: 7 processes: 2 remote cache hit, 5 internal.
INFO: Build completed successfully, 7 total actions
INFO: Running command line: bazel-bin/main
Hello, World!2

INFO: 7 processes: 2 remote cache hit, 5 internal.

[monaka]

Hmm... possiblly it depends on the configuration of K8s cluster ???
My Buildfarm is on AWS EKS.

I'll try this on my AzureAKS and Minikube.

[monaka]

Additional info:

I have two EKS clusters that are installed Buildfarm.
The one doesn't work RBE as I reported here.
But ... the other works well.

Even though I can't catch the difference between each cluster, I agree that Buildfarm works well on EKS.

I close this until I get some information for now. And I'll explore some more details.