Comments (2)
Hey Nathan,
Quick question: have you seen this document? To summarise, what I suggest that people do to get decent performance and consistency is to build a RAID-10 setup of systems that use LocalBlobAccess. Be sure to give that a try!
With regards to adding logic to let Redis be an index in front of S3: in theory we could support this, but I'm not a huge fan. Especially because it only works if Redis is set up without replication. As soon as you enable Redis replication, you'll discover that the index also doesn't provide the right consistency requirements you're looking for.
from bb-storage.
Awesome, that read was very useful. We have decided to revise our layout into:
- 2-3 large CAS instances with sharding to each other using local storage on Raid0 setup
- (we may add a mirroring set in the future depending on reliability needs)
- Worker nodes will use the
read_caching
withfast
set to S3 andslow
set to the CAS instances - Entry points will connect to one of the large CAS instances
- Workers will first try S3 and if fails to find it will download from one of the large CAS instances and upload it to S3
This should keep the bulk of the workload on S3 instead of the CAS. Granted all data must go through the CAS, but I think it's safe to assume most data will eventually be mirrored to S3 and since (from what I have read) S3 is limited to 25gbps per instance, we won't be suffering much on the worker nodes.
I think one thing that we may end up needing is the ability to verify the checksum of the S3 data and purge it if the checksum does not match. In early testing I have seen a couple cases of zero-byte data being downloaded from s3, but I have not yet dug deep into this issue.
from bb-storage.
Related Issues (20)
- Documentation needed for allowing cache access from different types of bazel clients HOT 1
- Missing shell in bb-storage docker image HOT 1
- Support for compression from buildbarn HOT 5
- Panic in local blockstore write HOT 3
- Doc Update: ISCC / AC storages are only compatible with local replicator HOT 1
- gRPC Client Certificate Refresh Interval is not respected
- Filesystem errors in bb-storage are recorded as "Unknown" in prometheus
- Feature request: Support RSA signed JWTs
- Feature request: Support JWKS for specifying JWT public keys HOT 4
- Failed to fetch file errors in "builds without the bytes" builds in a sharded setup HOT 7
- Failed to create authorization header parser for JWT authentication policy: Unsupported public key type HOT 3
- Cannot open raw block device provisioned by kubernetes when running as non root user HOT 3
- Support connection draining in kubernetes environments HOT 2
- Tunable LogLevels? HOT 3
- Is there any detailed description about config? HOT 6
- Creating buildbarn storage image doesn't work on bazel 7 HOT 2
- JWT: support ALB token format HOT 5
- Load server CA certificates from files for client configuration
- Publish images with arm64 HOT 4
- blake3 support HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bb-storage.