antidotedb / antidote

A planet scale, highly available, transactional database built on CRDT technology

AntidoteDB

Welcome to the Antidote repository, the reference platform of the SyncFree European Project and the LightKone European Project.

Description

AntidoteDB is a highly available geo-replicated key-value database. AntidoteDB provides features that help programmers to write correct applications while having the same performance and horizontal scalability as AP/NoSQL databases. Furthermore, AntidoteDB operations are based on the principle of synchronization-free execution by using Conflict-free replicated datatypes (CRDTs).

Features

CRDTs

High-level replicated data types that are designed to work correctly in the presence of concurrent updates and partial failures.

Highly Available Transactions

Traditional ACID transactions were built for single-machine deployments. On the one hand, it is expensive to implement ACID transactions in distributed deployments. On the other hand, highly-available transactions (HAT) provide strong consistency within a data center, but still perform well in geo-replicated deployments.

Geo-replication

Designed to run on multiple servers in locations distributed world-wide. It provides continuous functioning even when there are failures or network partition.

How to Use

You will find all information on the project website or the usage documentation.

Small tutorials on how to use Antidote can be found for Java and Jupyter Notebook.

Topics:

• Configuring Features of Antidote
• Benchmarking Antidote
• Deploying Antidote
  • Natively
  • Local Docker setup
  • Docker compose setups
  • Docker Swarm
  • Kubernetes
• Monitoring an Antidote instance or data center
• Protocol Buffer API
  • Erlang Client Repository
  • Java Client Repository
  • JavaScript Client Repository
  • Go Client Repository
  • Python Client Repository
  • REST Client Repository

Applications that use AntidoteDB:

• Calender App
• Antidote Web Shell

Contributing & Development

Antidote encourages open-source development. If you want to contribute, you can find all necessary information in the developer documentation To make yourself familiar with AntidoteDB, you can start by looking at good first issues.

antidote's Issues

Fix append_list_test and repfsm_error_handle_test

Neither of these tests is actually testing anything; append_list generates a partition after writing to the partitioned node, and is not exercising the behavior correctly.

Missing documentation in floppy_coord_sup.erl

Description of ClockSI protocol

Check correctness of protocols:
Are the data structures used the right ones?
Are messages all handled correctly?
Is fault-tolerance achieved? At which level?

Responsible: Alejandro

Fix logging_vnode:append_list

Missing documentation in inter_dc_*.erl

Missing documentation in floppy_console.erl

Replicated log: Generate unique opid after process restart

Now if rep_vnode restarts, opid is restarted from 1 which assigns duplicate opid to new updates which makes these updates invisible.

Write client for protocol buffer interface in Java

Adapt wallet to floppystore API

Modify wallet code to not use RPC's, but instead use the API.

Merge riak_test into Floppystore repo

Want to have tests the same location as code.

Remove clock si branches

Hello, after discussing with Deepthi, we agreed that we can remove the following bramches as they are inactive:

• clock-SI
• clock-SI_v2

The clock si functionallity is now provided in the causality branch, that has already a pull request to be merged to master. Then, in master, we should create some configuration mechanism in order to enable/disable the use of the protocol.

Test for ordering of log

Responsible: Manuel
Collaborator: Alejandro

Materializer slowness

The materializer has O(n) performance on operations, which needs to be addressed as a scalability concern.

Resolve dialyzer warnings on master

floppy_coord_fsm.erl:81: The call
floppy_rep_vnode:handleOp(IndexNode::integer(),pid(),Op::atom(),Key::any(),Param::any())
will never return since it differs in the 1st argument from the
success typing arguments: (maybe_improper_list({integer(),_},[] |
{integer(),_}) | {integer(),_},any(),any(),any(),any())
floppy_coord_fsm.erl:100: The call
floppy_rep_vnode:handleOp(IndexNode::integer(),pid(),Op::atom(),Key::any(),Param::any())
will never return since it differs in the 1st argument from the
success typing arguments: (maybe_improper_list({integer(),_},[] |
{integer(),_}) | {integer(),_},any(),any(),any(),any())

Code review of branches that will not be merged to master

Hello,

As we had agreed on the mumble meetings, branches that develop features like interDC replication or clockSI will not be merged to master in order to keep different "flavours" of floppystore, and make those features optional.
I wanted to ask when or how do we ask for code review for those branches. e.g. the "clock-SI_v2" branch and the "causality" branch have not been reviewed and it would be nice to have a review at this stage.

Merge for MV-Register

The merge function should fail and raise an error, even / especially if it is not implemented.

How is it related to merge_to? This looks like an implementation of merge.

Move to operation-based CRDTs

Restructure the log entries to generate down-stream operations for inter-DC replication

document api and its guarantees

so people can build something on top

Extend API with threshold read on log entry

Reads and writes should return operation id

cannot build, compilation of rebar test fails

Today, I've realised that the following error occurs when trying to build any branch (including master):

riak_test/floppystore/.rebar_plugins/rebar_test_plugin.erl:96: variable 'OutputDir' is unused
ERROR: riak_test_compile failed while processing /Users/alek/Dropbox/PhD/code/floppystore/floppystore: rebar_abort
make: *** [compile-riak-test] Error 1

Dialyzer warning for floppy_coord_sup:start_fsm

Dialyzer produces this error message:
floppy_rep_vnode.erl:51: Expression produces a value of type {'error',} | {'ok','undefined' | pid()} | {'ok','undefined' | pid(),}, but this value is unmatched

Similar for
floppy_rep_vnode.erl:67
floppy_rep_vnode.erl:100

This is due to calls such as:
floppy_coord_sup:start_fsm([self(), append, Key, Op])
receive
{ok,{_, Result}} ->
lager:info("Append completed!wn",[Result]),
{ok, Result}
after 5000 ->
lager:info("Append failed!~n"),
{error, timeout}
end.

It seems that the receive is trying to match another value?
The root of the cause seems to lie with starting child processes in floppy_coord_sup:start_fsm.

Update README.md

It needs info on:

• What is the purpose of floppystore? What does it do? How is it superior to Riak 2.0, which also has CRDT support?
• How is floppystore built? Including prerequisites.
• How are tests run?
• How are the built deployables deployed on a (set of) server(s)? What are the serer prerequisites?
• How is floppystore started and stopped?
• What operations can you run against floppystore? How do you run them?

It would be nice to also have documentation for:

• How is floppystore monitored. Is it running or not? Current or aggregated stats. BAckground processes.
• How can backups be made?
• CI setup.
• How to stress test floppystore? and possibly tune it?

append_failures_test is no longer passing

Figure out why.

Problems with the log and network partitions

There is bug caused by network partitions. The replication fsm only sends to the primary vnodes that are up. The preflist used to send to the vnodes is the one used for identifying the log in the logging_vnode. Since some vnodes might not be available, the preflist will not contain the down vnode; therefore, the preflist (composed by less than N ids) will not much any of the preflist that the vnode expects. In consequence the operation will fail.

Map composed by reference

Another test case for causality across partitions

interdcrepl: implement handoff

API for Floppystore on Protocol buffer

Replacing the rpc calls

Compare Map-by-reference and Map-by-embedding

Compare performance, message size, log size, ...
Re-use implementation from riak_DT/ riak_KV

Implement Multi-value-register CRDT

Add test
Setup demo
Should be an additional causality check

Too many files opened by a node

I've found an issue in the clock-SI + partitioned log branch (as I have merged them into clock-SI to support the "one log per preflist" feature).
In this branch, there are currently 4 files opened "per vnode" (as there is a logging_vnode that opens N; a log per partition, and a clock-SI vnode that opens one).

THE PROBLEM

When starting a single node cluster, all of these files are opened by a single unix process. I was getting the error:

[error] Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.1453.0> exit with reason bad return value: {stop,{file_error,"./data/1301649895747835411525156804137939564381064921088_log3",emfile}} in context child_terminated

Unix allows, by default, a maximum of 256 file descriptors opened per process, and this limit was exceeded when running a single node cluster. you can check that in your machine with "ulimit -n".

Solution

change (only for the current session) the maximum number of file descriptors allowed per process with:

ulimit -n new_limit

LEARNED:

• All erlang processes run in a single unix process (at least I didn't know that).

Refactor log

Break up log in different logs per partition -> N logs per vnode
Ensure that log is correctly sorted based on insertion order
Maybe choose an alternative to DETS for better performance, e.g. LevelDB?

Responsable: Manuel
Participating: Alejandro

Missing cache

Read latency increases when more writes are happening as each read is processing all log entries for the respective object.

Wrong usage of floppy_rep_vnode:handleOp

The signature of floppy_rep_vnode:handleOP differs from its usage in:
floppy_coord_fsm.erl:81
floppy_coord_fsm.erl:100

Causality test for Floppystore

Integrate single-key serializability

Participating: Alejandro

Preflist calculated from Key and LogId are not matching

log_utilities:get_preflist_from_key(Key)
and log_utilities:get_preflist_from_logid( get_logid_from_key(Key)) does not return same value.
Similarly, get_logid_from_partition(P) is not same as get_logid_from_key(Key), where
vnode with partition number P is head of preflist of Key. In many cases, we need to reach vnodes using partition number, without knowing which keys they are responsible for. How to fix this?

Try different persistent stores

So far we have just tried with dets. We should try with something else and see what performs better. I have abstracted the operations to the persistent store in the logging_vnode to ease the shifting to other persistent storage systems.

Resolve errors in walletapp

walletapp.erl:13: Expression produces a value of type {'error',_} | {'ok','undefined' | pid()} | {'ok','undefined' | pid(),_}, but this value is unmatched
walletclient.erl:10: Function start/1 has no local return
walletclient.erl:14: Function run/1 has no local return
walletclient.erl:19: The call walletclient:testvoucher(Key_voucher::any(),10,[]) will never return since it differs in the 2nd argument from the success typing arguments: (any(),0,[any()])
walletclient.erl:26: Expression produces a value of type 'undefined' | {integer(),integer(),integer()}, but this value is unmatched
walletclient.erl:34: Expression produces a value of type 'ok' | {'error',string()}, but this value is unmatched
walletclient.erl:36: The pattern {'error', Reason} can never match the type 'error' | number()
walletclient.erl:39: The variable _ can never match since previous clauses completely covered the type 'false'
walletclient.erl:44: Expression produces a value of type 'ok' | {'error',string()}, but this value is unmatched
walletclient.erl:46: The pattern {'error', Reason} can never match the type 'error' | number()
walletclient.erl:49: The variable _ can never match since previous clauses completely covered the type 'false'
walletclient.erl:52: Function testvoucher/3 has no local return
walletclient.erl:52: The pattern <_, 0, Result> can never match the type <_,10,[]>
walletclient.erl:56: Expression produces a value of type 'undefined' | {integer(),integer(),integer()}, but this value is unmatched
walletclient.erl:61: Function testadd/2 has no local return
walletclient.erl:63: Expression produces a value of type 'ok' | {'error',string()}, but this value is unmatched
walletclient.erl:65: The created fun has no local return
walletclient.erl:65: The call lists:usort(nonempty_improper_list(pos_integer(),number() | {'error',string()})) will never return since it differs in the 1st argument from the success typing arguments: ([any()])
walletclient.erl:65: Cons will produce an improper list since its 2nd argument is number() | {'error',string()}

Add licensing information

Apache 2.0 was to be used for the SyncFree github

Implement handoff and fold

Finish ClockSI implementation

Finish this.

interdcrepl: persist state for fault tolerance

Handle error messages in floppy_rep_fsm states

Add timeouts and error message handle for states waitread, waitappend. repairrest.

remove inactive branches

We should remove the branches that have already been merged to master and the ones that we don't expect to work on any longer.
Please, each branch creator report which branches he/she knows that we can delete now.

Fix quorum_unreachable.

When the replication coordinator generates a quorum_unreachable message, the coordinator for the request returns a normal timeout message, because the FSM fails to forward it back. Fix this.

Remove CamelCase and Fix Tabing

Change all CamelCase to snake_case, and while you're in the files, fix the tab indentation!

Specifying CRDT type when creation/ in each update

Currently, the type for a CRDT is specified only when reading.
However, generating downstream operations requires the type of CRDT, which is now hardcoded to riak_dt_pncounter.
Therefore, when using types other than counter, downstream op generator will throw exceptions.

Setup benchmark for Bounded Counters

Tools?
Demo for meeting in October

Setup benchmark for CRDTs on FloppyStore

Setup a benchmark for (unbounded) CRDT types, e.g. Counters
Do debugging and improving of Floppystore
Show throughput

Reading from log

Reading from the log in the logging vnode does not ensure operations are ordered given the op_id. This needs to be done to ensure the correct ordering of update operations.