Dealing with eventual consistency issues about bb-storage

[EdSchouten]

Hey Nathan,

Quick question: have you seen this document? To summarise, what I suggest that people do to get decent performance and consistency is to build a RAID-10 setup of systems that use LocalBlobAccess. Be sure to give that a try!

With regards to adding logic to let Redis be an index in front of S3: in theory we could support this, but I'm not a huge fan. Especially because it only works if Redis is set up without replication. As soon as you enable Redis replication, you'll discover that the index also doesn't provide the right consistency requirements you're looking for.

[allada]

Awesome, that read was very useful. We have decided to revise our layout into:

• 2-3 large CAS instances with sharding to each other using local storage on Raid0 setup
  • (we may add a mirroring set in the future depending on reliability needs)
• Worker nodes will use the read_caching with fast set to S3 and slow set to the CAS instances
• Entry points will connect to one of the large CAS instances
• Workers will first try S3 and if fails to find it will download from one of the large CAS instances and upload it to S3

This should keep the bulk of the workload on S3 instead of the CAS. Granted all data must go through the CAS, but I think it's safe to assume most data will eventually be mirrored to S3 and since (from what I have read) S3 is limited to 25gbps per instance, we won't be suffering much on the worker nodes.

I think one thing that we may end up needing is the ability to verify the checksum of the S3 data and purge it if the checksum does not match. In early testing I have seen a couple cases of zero-byte data being downloaded from s3, but I have not yet dug deep into this issue.