Comments (4)
Your worker must be dying at some point and deregistering/being expired from the backplane.
Present some logs from the worker, or verify why it is dropping off the map.
from bazel-buildfarm.
Thank you for your response. I couldn't find logs related to this issue from any workers.
(I have 2 workers.)
And I checked k8s pods and it seems that workers (and also Redis) don't restart.
How can I change the log-level for workers?
from bazel-buildfarm.
The error originates from this line:
At the time that it is thrown, no workers are present in the server's retention of the redis hash key "Workers_storage"
If you poll the values of that key in redis, you should see frequent updates (every 10s) to push to a time 30s into the future, by the workers that are registered. If there is a substantial oversubscription of a worker, these updates can be delayed and cause workers to be evicted due to expiration.
The question is: Are you transitioning to this state from a working one where the worker is available, or was it never accessible in the first place?
from bazel-buildfarm.
The question is: Are you transitioning to this state from a working one where the worker is available, or was it never accessible in the first place?
I suppose workers are available and accessible. Just sometimes lost.
Because files in the cache directory are updated continuously in the build stage.
And it seems like this issue may be resolved after fixing #1724 .
Just my guess but it may be occurred when buildfarm-shard and/or Redis are in heavy load.
Anyway this issue can be avoided. I close this.
Thank you for your help.
from bazel-buildfarm.
Related Issues (20)
- Fetch service does not properly handle missing content-length HOT 1
- Support Multiple Hashing Function
- Remote execution service executing processes locally HOT 4
- ERROR: Failed to query remote execution capabilities: UNAVAILABLE: io exception HOT 4
- Feasibility Analysis of Using Buildfarm for Large-Scale Development HOT 2
- Verbose Logging for Servers and Workers through Helm Chart
- Check logs during remote execution HOT 4
- Incorrect container port for Shard-worker in Helm template
- hardlinks in CAS leads to task failure in some cases HOT 2
- Fetch asset support for credential use
- [Bzlmod] No repository visible as '@maven' from main repository HOT 1
- Querying remote cache failed due to Missing Digest HOT 5
- Helm chart won't deploy workers because {ready,live}ness probes are using the wrong port
- When is the release? HOT 1
- Helm chart deployment storage issues
- FindMissingBlobs histogram has too few buckets
- CGroups v2 HOT 1
- RedisShardSubscription::stop is unsafe
- 无任务时redis CPU占用过高 HOT 2
- Transfer bazel-buildfarm to bazel-contrib HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bazel-buildfarm.