replikativ / datahike-server

Datahike remote system

replikativ

Project homepage

Changelog

Roadmap (suggestions)

0.3.0

• Investigate JS side integration of http://y-js.org/
• Investigate integration with similar systems, eg. IPFS pubsub
• Split middleware from replicated datatype implementations
• Improve network IO library kabel (Android support) [DONE]
• Move hashing into fetch middleware to simplify parallelization. [DONE]
• Experimental automatic Gossip protocol
• Experimental Snapshot Isolation
• Build reasonable small support libraries to partition application data for efficient client side consumption, Datomic and Datascript. Look into datsync etc.
• Add a monitoring interface as a cljs library with basic web views for applications to communicate their synching state to the user in a uniform way. [DONE]
• Introduce clojure.spec to stage/... API.

0.4.0

• Authentication with signed public-private key signatures
• Model some level of consistency between CRDTs, probably Snapshot Isolation, to compose CRDTs. (NMSI, Antidote, research)
• Implement more useful CRDTs (counter, vector-clock, ...) from techreview and other papers and ship by default.

0.5.0

• Use p2p block distribution similar to BitTorrent for immutable values (similar to blocks)
• support WebRTC for value distribution similar to BitTorrent
• Java bindings

Long-term (1.0.0)

• Encryption of transaction with CRDT key encrypted by userkeys, public key schema, explore pub/private key solutions. Maybe metadata signing can work (slowly) on a DHT?
• Distribute bandwidth between CRDTs.
• Negotiate middlewares with versioning.
• Implement diverse prototypes, from real-time to "big-data".

Contributors

• Konrad Kuehne
• Christian Weilbach

Support

If you would like to get some commercial support for replikativ, feel free to contact us at lambdaforge.

License

Copyright © 2013-2018 Christian Weilbach, Konrad Kühne

Distributed under the Eclipse Public License, the same as Clojure.

datahike-server's Issues

Switch from dockerhub to GH container registry

Docker Hub shuts down OSS-images without registering to some kind of thing or paying. We should just use GitHub container registry let's see how long we can rely on that...

continuous integration

pipeline for datahike-server

JSON support

Right now JSON is not fully working. This issue is high priority and will be worked on as soon as possible.

• figure out JSON to EDN serialisation
  • mapping of namespaces
  • keywords
  • symbols
  • datalog
• hammock for a good solution
• write an ADR if there are decisions to make
• implement the solution with reasonable testing

cleanup documentation

• add separate page to datahike.io with getting started instructions

Add metrics

• counter on datoms
• indices
• versions of backend and libs
• response times
• merge branch prometheus-metrics

/db should return more data

Currently /db only returns the current hash of the database. It should return also additional data like config and meta data

logging

basic logging with timbre

Add remote http commandline interface

It would be helpful to have a babashka client in the commandline that can talk to the server via a simple api.

server startup should log out current server config

It would be helpful to see the current server config maybe from log/info when starting the system.

REST API should have specs for responses

The data model for the api responses should be specified with clojure.spec both for validation and documentation

Bug: adding datahike config via env var results in exception

DATAHIKE_STORE_BACKEND=file java -jar target/datahike-server-0.1.81-standalone.jar                       
                                                          
2022-06-02 22:42:36.609:INFO::main: Logging initialized @4314ms to org.eclipse.jetty.util.log.StdErrLog
2022-06-02T20:42:36.733Z infinity DEBUG [datahike-server.config:54] - Loading config
2022-06-02T20:42:36.742Z infinity INFO [datahike-server.config:25] - No config file found at  resources/config.edn
2022-06-02T20:42:36.743Z infinity DEBUG [datahike-server.database:34] - Connecting to databases with config:  {:server {:port 3000, :join? false, :loglevel :info, :dev-mode false}, :databases nil}
Exception in thread "main" java.lang.RuntimeException: could not start [#'datahike-server.database/conns] due to
	at mount.core$up$fn__216.invoke(core.cljc:80)
	at mount.core$up.invokeStatic(core.cljc:80)
	at mount.core$up.invoke(core.cljc:78)
	at mount.core$bring.invokeStatic(core.cljc:247)
	at mount.core$bring.invoke(core.cljc:239)
	at mount.core$start.invokeStatic(core.cljc:289)
	at mount.core$start.doInvoke(core.cljc:281)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at datahike_server.core$_main.invokeStatic(core.clj:16)
	at datahike_server.core$_main.doInvoke(core.clj:15)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at datahike_server.core.main(Unknown Source)
Caused by: java.lang.NullPointerException
	at konserve.filestore$check_and_create_folder.invokeStatic(filestore.clj:398)
	at konserve.filestore$check_and_create_folder.invoke(filestore.clj:395)
	at konserve.filestore$new_fs_store.invokeStatic(filestore.clj:442)
	at konserve.filestore$new_fs_store.doInvoke(filestore.clj:435)
	at clojure.lang.RestFn.invoke(RestFn.java:410)
	at datahike.store$fn__19319.invokeStatic(store.cljc:108)
	at datahike.store$fn__19319.invoke(store.cljc:107)
	at clojure.lang.MultiFn.invoke(MultiFn.java:229)
	at datahike.connector$fn__24676.invokeStatic(connector.cljc:125)
	at datahike.connector$fn__24676.invoke(connector.cljc:107)
	at datahike.connector$fn__24633$G__24617__24638.invoke(connector.cljc:101)
	at datahike.connector$database_exists_QMARK_.invokeStatic(connector.cljc:248)
	at datahike.connector$database_exists_QMARK_.invoke(connector.cljc:246)
	at datahike_server.database$init_connections.invokeStatic(database.clj:10)
	at datahike_server.database$init_connections.invoke(database.clj:8)
	at datahike_server.database$fn__24782.invokeStatic(database.clj:35)
	at datahike_server.database$fn__24782.invoke(database.clj:32)
	at mount.core$record_BANG_.invokeStatic(core.cljc:74)
	at mount.core$record_BANG_.invoke(core.cljc:73)
	at mount.core$up$fn__216.invoke(core.cljc:81)
	... 13 more

Add authentication by token

Make authentication by token possible.

Add healthcheck livenesscheck

dockerfile

basic dockerfile

Switch CI to clojure tools and GitHub-flow

• Switch to tools deps
• Use tools build
• Switch CI to GitHub-flow
• Update docs
• Switch to main branch
• Update deps

API documentation missing

So far only the server setup seems to be documented.
Some documentation on how to use the resulting API would make it easier for developers to program clients for datahike and for people that want to use the REST interface directly.

Use string as token not keyword

The token should accept special chars and therefore should not be transformed to keyword.

Add history support

• hammock the history support on how to provide point-in-time to datahike-server
  • header? tx-id or date-time?

[Bug] database name not available in latest Datahike version

There was a change lately to make Datahike distributed but now connect does not return name anymore. We need to use store/store-identity to identify databases to query against.

Bug is here: https://github.com/replikativ/datahike-server/blob/main/src/clj/datahike_server/database.clj#L28-L29

Archive datahike-server repo

Can I archive this repo and point to the new datahike server from the main repo? @whilo @alekcz

Make datahike-server compliant with datahike

• db-record-fns
• check what is missing...

azure deploy

azure container instance on button click

Deciding on License

Since cloud providers love to offer opensource software as their own service it might be an idea to switch to a license that does not permit usage as a service offering.
Article discussing that: https://onezero.medium.com/open-source-betrayed-industry-leaders-accuse-amazon-of-playing-a-rigged-game-with-aws-67177bc748b7

pull with db-tx fails with valAt/2 is not supported on AsOfDB

http :3000/db db-name:foo returns:

{
    "tx": 536870915
}

echo '{:selector [:age :name] :eid 4}' | http POST :3000/pull db-name:foo content-type:application/edn db-tx:536870915 returns:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /pull. Reason:
<pre>    valAt/2 is not supported on AsOfDB</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

But it should return something like this:

{
    "age": 40,
    "name": "Charlie"
}

the configuration should have customizable file paths and be dependent on the environment

Currently the tests always overwrite my local configuration. The current environment should set its configuration file target and configurations should live in independent files.

How to specify backend support

• The server should be able to start up without adding anything by hand. What should be the default backend?
• How to make sure that backends work with the current server? Release with same version number? monorepo?
• Where to list the supported backends? README?

configuration for datahike-server

datahike-server needs a way to be configured

• port
• interface
• join?

[Chore] Update dependencies to latest

Especially datahike has evolved and this means a lot of work to update. Mainly the tests will have problems.

Create an openapi spec

We want to make Datahike server a spec-first implementation. So we need a spec over the whole api.

• versioned
• yml
• CI with linting

Use openapi-spec to generate api

• Probably navi
• reitit with navi should be able to create api from openapi spec

Transparent RPCs for datahike.api

The server should provide the database management functions such as create-database and delete-database and datahike.api invocations should be transparently routeable to a remote server that way. A separate naming scheme for databases is not needed, instead the full configuration or store-identity can be send to address databases. This allows to also route transact and query calls to a server without additional information.

Server broken with latest Datahike version

I updated the deps for datahike-server, built it and tried to run it with dev-mode. It fails with:

2023-03-22 17:09:44.714:INFO::main: Logging initialized @2757ms to org.eclipse.jetty.util.log.StdErrLog
2023-03-22T16:09:44.829Z Zockolette DEBUG [datahike-server.config:33] - Loading config
2023-03-22T16:09:44.842Z Zockolette INFO [datahike-server.config:27] - No config file found at  resources/config.edn
2023-03-22T16:09:44.843Z Zockolette DEBUG [datahike-server.database:9] - Connecting to databases with config:  {:server {:port 3000, :join? false, :loglevel :info, :dev-mode true}, :databases nil}
2023-03-22T16:09:44.856Z Zockolette INFO [datahike-server.database:12] - Creating database...
Exception in thread "main" java.lang.RuntimeException: could not start [#'datahike-server.database/conns] due to
        at mount.core$up$fn__216.invoke(core.cljc:80)
        at mount.core$up.invokeStatic(core.cljc:80)
        at mount.core$up.invoke(core.cljc:78)
        at mount.core$bring.invokeStatic(core.cljc:247)
        at mount.core$bring.invoke(core.cljc:239)
        at mount.core$start.invokeStatic(core.cljc:289)
        at mount.core$start.doInvoke(core.cljc:281)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at datahike_server.core$_main.invokeStatic(core.clj:10)
        at datahike_server.core$_main.doInvoke(core.clj:9)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at datahike_server.core.main(Unknown Source)
Caused by: clojure.lang.ArityException: Wrong number of args (0) passed to: datahike.writer/backend-dispatch
        at clojure.lang.AFn.throwArity(AFn.java:429)
        at clojure.lang.RestFn.invoke(RestFn.java:399)
        at clojure.lang.MultiFn.invoke(MultiFn.java:225)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.core$apply.invoke(core.clj:662)
        at datahike.api$create_database.invokeStatic(api.cljc:114)
        at datahike.api$create_database.doInvoke(api.cljc:113)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at datahike_server.database$init_connections.invokeStatic(database.clj:13)
        at datahike_server.database$init_connections.invoke(database.clj:8)
        at datahike_server.database$fn__34091.invokeStatic(database.clj:38)
        at datahike_server.database$fn__34091.invoke(database.clj:37)
        at mount.core$record_BANG_.invokeStatic(core.cljc:74)
        at mount.core$record_BANG_.invoke(core.cljc:73)
        at mount.core$up$fn__216.invoke(core.cljc:81)
        ... 13 more

Better error information on datahike error

When an exception is thrown out of the Datahike code, e.g. due to an erroneous query formulation, the server responds with status 500 and only information about error type and class in its body.
It would be nice though to have more information about why the error occurred, i.e. at least the exception info if it is an exception.