theanalyst / are-we-consistent-yet

Eventual consistency of object stores

Are We Consistent Yet?

Observed and documented eventual consistency of object stores, e.g., Amazon S3, OpenStack Swift.

What is eventual consistency?

Traditional systems provide strong consistency, where clients can immediately view updates. Some distributed systems relax their consistency model to allow greater availability or better performance. Eventual consistency manifests itself to clients as stale views of data.

Observed consistency

We ran a test in which we wrote (i.e., create, update, delete) an object and then attempted to read the object. Across many trials, we count the number of times the object was not immediately available. When the object is not immediately found, it's an occurrence of observed eventual consistency.

Observed instances of eventual consistency with a number of operations during tests around 17 December 2014 with a 1-byte object size:

Provider	read after create	read after delete	read after overwrite	list after create	list after delete	number of operations
Amazon S3 (us-standard)	*	*	*	16	*	100,000
Amazon S3 (us-standard†)	16	8	12	8	3	1,000
Amazon S3 (us-west)	-	1	1	*	*	100,000
Ceph (DreamObjects)	-	-	-	-	-	1,000
Google Cloud Storage	-	-	-	2	2	1,000
Microsoft Azure Storage	-	-	-	-	-	1,000
OpenStack Swift (Rackspace)	-	17	3	30	20	1,000

Legend:

• • zero observed instances
• • zero observed instances but expect non-zero
• † writing to N. Virginia and reading from Pacific Northwest

Documented consistency

• Amazon S3 - buckets in the US Standard region provide eventual consistency. Buckets in all other regions provide read-after-write consistency for PUTs of new objects and eventual consistency for overwrite PUTs and DELETEs.
• Ceph - provides well-defined safety semantics and strong consistency guarantees.
• Google Cloud Storage - provides strong global consistency for all read-after-write, read-after-update, and read-after-delete operations, including both data and metadata. List operations are eventually consistent.
• Microsoft Azure Storage - was designed to embrace a strong consistency model which guarantees that when the Storage service commits a data insert or update operation all further accesses to that data will see the latest update.
• OpenStack Swift - For example, suppose a container server is under load and a new object is put in to the system. The object will be immediately available for reads as soon as the proxy server responds to the client with success. However, the container server did not update the object listing, and so the update would be queued for a later update. Container listings, therefore, may not immediately contain the object. Additional reference.

References

• Apache jclouds - provides object storage support
• A Middleware Guaranteeing Client-Centric Consistency on Top of Eventually Consistent Datastores
• Benchmarking Eventual Consistency: Lessons Learned from Long-Term Experimental Studies
• Eventual Consistency: How soon is eventual?
• Eventual Consistency Today: Limitations, Extensions, and Beyond
• Probabilistically Bounded Staleness - includes discussion of Riak and Cassandra