replikativ / hitchhiker-tree

Functional, persistent, off-heap, high performance data structure

replikativ

Project homepage

Changelog

Roadmap (suggestions)

0.3.0

• Investigate JS side integration of http://y-js.org/
• Investigate integration with similar systems, eg. IPFS pubsub
• Split middleware from replicated datatype implementations
• Improve network IO library kabel (Android support) [DONE]
• Move hashing into fetch middleware to simplify parallelization. [DONE]
• Experimental automatic Gossip protocol
• Experimental Snapshot Isolation
• Build reasonable small support libraries to partition application data for efficient client side consumption, Datomic and Datascript. Look into datsync etc.
• Add a monitoring interface as a cljs library with basic web views for applications to communicate their synching state to the user in a uniform way. [DONE]
• Introduce clojure.spec to stage/... API.

0.4.0

• Authentication with signed public-private key signatures
• Model some level of consistency between CRDTs, probably Snapshot Isolation, to compose CRDTs. (NMSI, Antidote, research)
• Implement more useful CRDTs (counter, vector-clock, ...) from techreview and other papers and ship by default.

0.5.0

• Use p2p block distribution similar to BitTorrent for immutable values (similar to blocks)
• support WebRTC for value distribution similar to BitTorrent
• Java bindings

Long-term (1.0.0)

• Encryption of transaction with CRDT key encrypted by userkeys, public key schema, explore pub/private key solutions. Maybe metadata signing can work (slowly) on a DHT?
• Distribute bandwidth between CRDTs.
• Negotiate middlewares with versioning.
• Implement diverse prototypes, from real-time to "big-data".

Contributors

• Konrad Kuehne
• Christian Weilbach

Support

If you would like to get some commercial support for replikativ, feel free to contact us at lambdaforge.

License

Copyright © 2013-2018 Christian Weilbach, Konrad Kühne

Distributed under the Eclipse Public License, the same as Clojure.

hitchhiker-tree's Issues

Reactive cljs tests

Currently the ClojureScript code is untested (and very likely superficially broken).

Add mark phase for konserve sweep garbage collector.

Add a mark phase for the new konserve garbage collection support.

Support synchronous konserve IO

Cleanup and factorize code base

The code base is currently not very accessible and badly factorized. We should go through all the comments and documentation of the codebase and collect it. We should also move all IO related code (including serialization) into separate namespaces and ideally break this out into separate repositories.

Support for built-in tagged elements (uuid and inst)

hitchhiker.tree.core/order-on-edn-types does not support uuids or insts.
This prevents uuids from being used in lookup refs in datahike.

add reverse iterator

design upsert operation for datoms

AssertionError occurs irregularly in forward-iterator

Running datahike-benchmark with
lein run -e
sometimes throws following AssertionError:

clojure.lang.ExceptionInfo: Assert failed: (tree/data-node? (peek path))
{:original-error #error {
:cause "Assert failed: (tree/data-node? (peek path))"
:via
[{:type java.lang.AssertionError
:message "Assert failed: (tree/data-node? (peek path))"
:at [hitchhiker.tree.messaging$forward_iterator invokeStatic "messaging.cljc" 216]}]
:trace
[[hitchhiker.tree.messaging$forward_iterator invokeStatic "messaging.cljc" 216]
[hitchhiker.tree.messaging$forward_iterator invoke "messaging.cljc" 212]
[hitchhiker.tree.messaging$forward_iterator$fn__11861 invoke "messaging.cljc" 220]
[clojure.lang.LazySeq sval "LazySeq.java" 42]
[clojure.lang.LazySeq seq "LazySeq.java" 58]
[clojure.lang.Cons next "Cons.java" 39]
[clojure.lang.RT next "RT.java" 713]
[clojure.lang.SeqIterator hasNext "SeqIterator.java" 41]
[clojure.lang.TransformerIterator step "TransformerIterator.java" 74]
[clojure.lang.TransformerIterator hasNext "TransformerIterator.java" 97]
[clojure.lang.RT$4 invoke "RT.java" 522]
[clojure.lang.LazySeq sval "LazySeq.java" 42]
[clojure.lang.LazySeq seq "LazySeq.java" 51]
[clojure.lang.ChunkedCons chunkedNext "ChunkedCons.java" 59]
[clojure.lang.ChunkedCons next "ChunkedCons.java" 43]
[clojure.lang.PersistentVector create "PersistentVector.java" 79]
[clojure.lang.LazilyPersistentVector create "LazilyPersistentVector.java" 44]
[clojure.core$vec invokeStatic "core.clj" 377]
[clojure.core$vec invoke "core.clj" 367]
[datahike.db$init_max_eid invokeStatic "db.cljc" 734]
[datahike.db$init_max_eid invoke "db.cljc" 727]
[datahike.connector$eval23844$fn__23847 invoke "connector.cljc" 139]
[datahike.connector$eval23782$fn__23805$G__23767__23810 invoke "connector.cljc" 83]
[datahike.connector$connect invokeStatic "connector.cljc" 229]
[datahike.connector$connect invoke "connector.cljc" 195]
[datahike_benchmark.db.datahike$eval31600$fn__31601 invoke "datahike.clj" 8]
[clojure.lang.MultiFn invoke "MultiFn.java" 234]
[datahike_benchmark.db.api$connect invokeStatic "api.clj" 16]
[datahike_benchmark.db.api$connect invoke "api.clj" 16]
[datahike_benchmark.measure.function_specific$eval31761$fn__31762$fn__31763 invoke "function_specific.clj" 46]
[datahike_benchmark.measure.api$eval32752$fn__32753$iter__32754__32758$fn__32759$fn__32760$fn__32761$fn__32765 invoke "api.clj" 65]
[datahike_benchmark.measure.api$get_time_with_res$fn__32745 invoke "api.clj" 53]
[datahike_benchmark.measure.api$get_time_with_res invokeStatic "api.clj" 53]
[datahike_benchmark.measure.api$get_time_with_res invoke "api.clj" 50]
[datahike_benchmark.measure.api$eval32752$fn__32753$iter__32754__32758$fn__32759$fn__32760$fn__32761 invoke "api.clj" 65]
[datahike_benchmark.measure.api$eval32752$fn__32753$iter__32754__32758$fn__32759$fn__32760 invoke "api.clj" 64]
[datahike_benchmark.measure.api$eval32752$fn__32753$iter__32754__32758$fn__32759 invoke "api.clj" 63]
[clojure.lang.LazySeq sval "LazySeq.java" 42]
[clojure.lang.LazySeq seq "LazySeq.java" 51]
[clojure.lang.RT seq "RT.java" 535]
[clojure.lang.LazilyPersistentVector create "LazilyPersistentVector.java" 44]
[clojure.core$vec invokeStatic "core.clj" 377]
[clojure.core$vec invoke "core.clj" 367]
[datahike_benchmark.measure.api$eval32752$fn__32753 invoke "api.clj" 63]
[clojure.lang.MultiFn invoke "MultiFn.java" 272]
[clojure.lang.AFn applyToHelper "AFn.java" 186]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 671]
[clojure.core$partial$fn__5845 doInvoke "core.clj" 2643]
[clojure.lang.RestFn invoke "RestFn.java" 421]
[datahike_benchmark.bench.connection$run_combinations$iter__32853__32859$fn__32860$iter__32855__32861$fn__32862$fn__32863$fn__32864 invoke "connection.clj" 25]
[datahike_benchmark.bench.connection$run_combinations$iter__32853__32859$fn__32860$iter__32855__32861$fn__32862$fn__32863 invoke "connection.clj" 20]
[datahike_benchmark.bench.connection$run_combinations$iter__32853__32859$fn__32860$iter__32855__32861$fn__32862 invoke "connection.clj" 13]
[clojure.lang.LazySeq sval "LazySeq.java" 42]
[clojure.lang.LazySeq seq "LazySeq.java" 51]
[clojure.lang.RT seq "RT.java" 535]
[clojure.core$seq__5402 invokeStatic "core.clj" 137]
[clojure.core$seq__5402 invoke "core.clj" 137]
[datahike_benchmark.bench.connection$run_combinations$iter__32853__32859$fn__32860 invoke "connection.clj" 13]
[clojure.lang.LazySeq sval "LazySeq.java" 42]
[clojure.lang.LazySeq seq "LazySeq.java" 58]
[clojure.lang.ChunkedCons chunkedNext "ChunkedCons.java" 59]
[clojure.lang.ChunkedCons next "ChunkedCons.java" 43]
[clojure.lang.RT next "RT.java" 713]
[clojure.core$next__5386 invokeStatic "core.clj" 64]
[clojure.core$dorun invokeStatic "core.clj" 3142]
[clojure.core$doall invokeStatic "core.clj" 3148]
[clojure.core$doall invoke "core.clj" 3148]
[datahike_benchmark.bench.connection$run_combinations invokeStatic "connection.clj" 13]
[datahike_benchmark.bench.connection$run_combinations invoke "connection.clj" 9]
[datahike_benchmark.bench.connection$eval32883$fn__32884 invoke "connection.clj" 43]
[clojure.lang.MultiFn invoke "MultiFn.java" 244]
[datahike_benchmark.bench.api$bench invokeStatic "api.clj" 18]
[datahike_benchmark.bench.api$bench invoke "api.clj" 10]
[datahike_benchmark.core$_main$iter__34078__34082$fn__34083$fn__34084 invoke "core.clj" 86]
[datahike_benchmark.core$_main$iter__34078__34082$fn__34083 invoke "core.clj" 82]
[clojure.lang.LazySeq sval "LazySeq.java" 42]
[clojure.lang.LazySeq seq "LazySeq.java" 51]
[clojure.lang.RT seq "RT.java" 535]
[clojure.core$seq__5402 invokeStatic "core.clj" 137]
[clojure.core$dorun invokeStatic "core.clj" 3133]
[clojure.core$doall invokeStatic "core.clj" 3148]
[clojure.core$doall invoke "core.clj" 3148]
[datahike_benchmark.core$_main invokeStatic "core.clj" 82]
[datahike_benchmark.core$_main doInvoke "core.clj" 70]
[clojure.lang.RestFn invoke "RestFn.java" 397]
[clojure.lang.Var invoke "Var.java" 380]
[user$eval140 invokeStatic "form-init15868104348883078227.clj" 1]
[user$eval140 invoke "form-init15868104348883078227.clj" 1]
[clojure.lang.Compiler eval "Compiler.java" 7177]
[clojure.lang.Compiler eval "Compiler.java" 7167]
[clojure.lang.Compiler load "Compiler.java" 7636]
[clojure.lang.Compiler loadFile "Compiler.java" 7574]
[clojure.main$load_script invokeStatic "main.clj" 475]
[clojure.main$init_opt invokeStatic "main.clj" 477]
[clojure.main$init_opt invoke "main.clj" 477]
[clojure.main$initialize invokeStatic "main.clj" 508]
[clojure.main$null_opt invokeStatic "main.clj" 542]
[clojure.main$null_opt invoke "main.clj" 539]
[clojure.main$main invokeStatic "main.clj" 664]
[clojure.main$main doInvoke "main.clj" 616]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.main main "main.java" 40]]}, :iteration 1}
at datahike_benchmark.measure.api$time_error.invokeStatic (api.clj:30)
datahike_benchmark.measure.api$time_error.invoke (api.clj:29)
datahike_benchmark.measure.api$eval32752$fn__32753$iter__32754__32758$fn__32759$fn__32760$fn__32761.invoke (api.clj:69)
datahike_benchmark.measure.api$eval32752$fn__32753$iter__32754__32758$fn__32759$fn__32760.invoke (api.clj:64)
datahike_benchmark.measure.api$eval32752$fn__32753$iter__32754__32758$fn__32759.invoke (api.clj:63)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:51)
clojure.lang.RT.seq (RT.java:535)
clojure.lang.LazilyPersistentVector.create (LazilyPersistentVector.java:44)
clojure.core$vec.invokeStatic (core.clj:377)
clojure.core$vec.invoke (core.clj:367)
datahike_benchmark.measure.api$eval32752$fn__32753.invoke (api.clj:63)
clojure.lang.MultiFn.invoke (MultiFn.java:272)
clojure.lang.AFn.applyToHelper (AFn.java:186)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.core$apply.invokeStatic (core.clj:671)
clojure.core$partial$fn__5845.doInvoke (core.clj:2643)
clojure.lang.RestFn.invoke (RestFn.java:421)
datahike_benchmark.bench.connection$run_combinations$iter__32853__32859$fn__32860$iter__32855__32861$fn__32862$fn__32863$fn__32864.invoke (connection.clj:25)
datahike_benchmark.bench.connection$run_combinations$iter__32853__32859$fn__32860$iter__32855__32861$fn__32862$fn__32863.invoke (connection.clj:20)
datahike_benchmark.bench.connection$run_combinations$iter__32853__32859$fn__32860$iter__32855__32861$fn__32862.invoke (connection.clj:13)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:51)
clojure.lang.RT.seq (RT.java:535)
clojure.core$seq__5402.invokeStatic (core.clj:137)
clojure.core/seq (core.clj:137)
datahike_benchmark.bench.connection$run_combinations$iter__32853__32859$fn__32860.invoke (connection.clj:13)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:58)
clojure.lang.ChunkedCons.chunkedNext (ChunkedCons.java:59)
clojure.lang.ChunkedCons.next (ChunkedCons.java:43)
clojure.lang.RT.next (RT.java:713)
clojure.core$next__5386.invokeStatic (core.clj:64)
clojure.core$dorun.invokeStatic (core.clj:3142)
clojure.core$doall.invokeStatic (core.clj:3148)
clojure.core$doall.invoke (core.clj:3148)
datahike_benchmark.bench.connection$run_combinations.invokeStatic (connection.clj:13)
datahike_benchmark.bench.connection$run_combinations.invoke (connection.clj:9)
datahike_benchmark.bench.connection$eval32883$fn__32884.invoke (connection.clj:43)
clojure.lang.MultiFn.invoke (MultiFn.java:244)
datahike_benchmark.bench.api$bench.invokeStatic (api.clj:18)
datahike_benchmark.bench.api$bench.invoke (api.clj:10)
datahike_benchmark.core$_main$iter__34078__34082$fn__34083$fn__34084.invoke (core.clj:86)
datahike_benchmark.core$_main$iter__34078__34082$fn__34083.invoke (core.clj:82)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:51)
clojure.lang.RT.seq (RT.java:535)
clojure.core$seq__5402.invokeStatic (core.clj:137)
clojure.core$dorun.invokeStatic (core.clj:3133)
clojure.core$doall.invokeStatic (core.clj:3148)
clojure.core$doall.invoke (core.clj:3148)
datahike_benchmark.core$_main.invokeStatic (core.clj:82)
datahike_benchmark.core$_main.doInvoke (core.clj:70)
clojure.lang.RestFn.invoke (RestFn.java:397)
clojure.lang.Var.invoke (Var.java:380)
user$eval140.invokeStatic (form-init15868104348883078227.clj:1)
user$eval140.invoke (form-init15868104348883078227.clj:1)
clojure.lang.Compiler.eval (Compiler.java:7177)
clojure.lang.Compiler.eval (Compiler.java:7167)
clojure.lang.Compiler.load (Compiler.java:7636)
clojure.lang.Compiler.loadFile (Compiler.java:7574)
clojure.main$load_script.invokeStatic (main.clj:475)
clojure.main$init_opt.invokeStatic (main.clj:477)
clojure.main$init_opt.invoke (main.clj:477)
clojure.main$initialize.invokeStatic (main.clj:508)
clojure.main$null_opt.invokeStatic (main.clj:542)
clojure.main$null_opt.invoke (main.clj:539)
clojure.main$main.invokeStatic (main.clj:664)
clojure.main$main.doInvoke (main.clj:616)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.lang.Var.applyTo (Var.java:705)
clojure.main.main (main.java:40)