rivine / recordchain

like a blockchain but then based on records like in a db

Rivine

Blockchain technology for creating custom chains.

Whitepaper

You can reach out to us for any development-related questions on our public Telegram group.

Repository has been moved

This repository is no longer active. Rivine has been moved to a new organization and its development is continued in its new home at https://github.com/threefoldtech/rivine.

Usage

This project comes with 2 binaries, rivined and rivinec. rivined is a background service, or "daemon," that runs the Rivine protocol, and rivinec is a client that is used to interact with rivined. rivined exposes an HTTP API on 'localhost:23110' which can be used to interact with the daemon. Documentation on the API can be found in doc/API.md.

rivined and rivinec are run via command prompt. On Windows, you can just double- click rivined.exe if you don't need to specify any command-line arguments. Otherwise, navigate to the rivine folder and click File->Open command prompt. Then, start the rivined service by entering rivined and pressing Enter. The command prompt may appear to freeze; this means rivined is waiting for requests. Windows users may see a warning from the Windows Firewall; be sure to check both boxes ("Private networks" and "Public networks") and click "Allow access." You can now run rivinec in a separate command prompt to interact with rivined.

Building From Source

To build from source, Go 1.9 or higher must be installed on the system (older versions of GO might work, but we don't support it). Then simply use go get:

go get -u github.com/rivine/rivine/cmd/...

This will download the Rivine repo to your $GOPATH/src folder, and install the rivined and rivinec binaries in your $GOPATH/bin folder.

To stay up-to-date, run the previous go get command again. Alternatively, you can use the Dockerfile provided in this repo. Run docker build -t rivine . to build and docker run --name rivine rivine to start the daemon. Running the client can be done with docker run -it rivine rivinec. Add client commands just like you would calling rivinec normally (like docker run -it rivine rivinec wallet transactions).

Supporting a Rivine Wallet in a Light Client

Should you wish to support Rivine, or more likely a Rivine-based wallet in a(n) existing/new light client, it is recommended that you read /doc/transactions/light_wallet.md as a starting point.

This document references to (most of the) other documentation you'll need in order to develop your own light wallet for a Rivine-Protocol-based blockchain.

Troubleshooting

• I can't connect to more than 8 peers.

  Once Rivine has connected to 8 peers, it will stop trying to form new connections, but it will still accept incoming connection requests (up to 128 total peers). However, if you are behind a firewall, you will not be able to accept incoming connections. You must configure your firewall to allow Rivine connections by forwarding your ports. By default, Rivine communicates on port 23112. The specific instructions for forwarding a port vary by router. For more information, consult this guide.

  Rivine currently has support for UPnP. While not all routers support UPnP, a majority of users should have their ports automatically forwarded by UPnP.

• I loaded a seed, but my coins don't appear yet.

  After loading your seed, you will need to stop and restart your daemon. If after that the coins still haven't appeared (and you are sure they should be there), you can try rivinec wallet addresses to force load some more addresses.

  Note that when you recover a wallet from an existing seed, using wallet recover, effectively creating a new wallet using an existing set, your addresses will already be loaded when unlocking it.

• My daemon exits imediately upon startup, giving the error: error opening consensus database: invalid argument:

  Make sure that the file system you are using for your daemon's persistent directory supports features such as mmap and flock. Shared folders for example, as is common in virtual environments or containers, might not support mmap and therefore will fail to function. The scenario in this example can easily be resolved by making sure the persistent directory isn't located within a shared folder.

• My node can be out of sync with the network for hours, if not days.

  Do you have multiple nodes —which includes the out-of-sync node— running on a single external IP using the same port within the same blockchain network? If the answer is yes, you should ensure that all those nodes are using unique ports for the RPC address.

  If not already, you will run into problems sooner or later when running multiple nodes using the same external IP address and RPC address port within the same blockchain network.

recordchain's Issues

potential security vulnerability with pycrypto (<= 2.6.1)

GitHub Security alerts found a potential security vulnerability in this repository:

Known moderate severity security vulnerability detected in pycrypto <=2.6.1
defined in requirements.txt.

Review vulnerable dependency: https://github.com/rivine/recordchain/network/dependencies

Need to check that required parameters have been sent on sell and buy orders

Steps

• Send buy order with price max only
• client.order_book.add_buy_order(price_max='10 ETH')

Issue

Order will be created and take id although it's wrong order

Matching algorithm for buying vs selling orders

Expand Jumpscale currency framework to include more digital coins

Currently we have a numeric data type in Jumpscale that can understand currency

In [11]: x =j.data.types.numeric.fromString('10 USD')

In [12]: j.data.types.numeric.bytes2cur(x)
Out[12]: 10.0

We want to expand it to understand / convert and fetch latest values for (BTC, ETH, XRP, TFT)

bcdb: data is a reserved word that is not usable in a schema definition

Implement keep allive

each node will send messages ( http calls with jwt and ip of node ) to the orderbook.
When the orderbook doesn’t receive these messages from the node, the orders will be deleted.
A node will always send his orders when a connection with the orderbook is created.

implement test in zdb client

/code/github/rivine/recordchain/JumpScale9RecordChain/clients/zdb/ZDBClient.py

because new implementation now of index (in zdb directly)

Need parameters validation

• Need to validate the following parameters have that valid currency on sell, buy and update orders
  • currency_to_buy
  • currency_to_sell
  • currency_mine
  • currency_accept

create web UI for orderbook

Need Apis for tracking transactions

need transaction APIs to can get

• My transactions.
• Filter transactions with ID.

list orders doesn't work

issue

try to list orders doesn't work as shown if the list doesn't empty

In [50]:  x.order_book.list_my_buy_orders()
---------------------------------------------------------------------------
ResponseError                             Traceback (most recent call last)
/usr/local/bin/js9 in <module>()
----> 1 x.order_book.list_my_buy_orders()

/opt/code/github/rivine/recordchain/JumpScale9RecordChain/apps/orderbook/client/cmds_orderbook_order_book.py in list_my_buy_orders(self, sortby, desc)
     54         args.desc = desc
     55 
---> 56         res = self._redis.execute_command("order_book.list_my_buy_orders",args.data)
     57 
     58 

/usr/local/lib/python3.5/dist-packages/redis/client.py in execute_command(self, *args, **options)
    666         try:
    667             connection.send_command(*args)
--> 668             return self.parse_response(connection, command_name, **options)
    669         except (ConnectionError, TimeoutError) as e:
    670             connection.disconnect()

/usr/local/lib/python3.5/dist-packages/redis/client.py in parse_response(self, connection, command_name, **options)
    678     def parse_response(self, connection, command_name, **options):
    679         "Parses a response from the Redis server"
--> 680         response = connection.read_response()
    681         if command_name in self.response_callbacks:
    682             return self.response_callbacks[command_name](response, **options)

/usr/local/lib/python3.5/dist-packages/redis/connection.py in read_response(self)
    627             raise
    628         if isinstance(response, ResponseError):
--> 629             raise response
    630         return response
    631 

ResponseError: ('unorderable types: NoneType() < NoneType()',)

implement / fix 2 tests on j.servers.gedisexample

js9 'j.servers.gedisexample.test()'
js9 'j.servers.gedisexample.test_ssl()'

generate server python & capnp

store in this repo
also script

more readable docs, check in readme's

dont do docu like this better

is more readable & can be copy pasted right away

Enable SSL support for gedis2

Review orderbook concepts and doc/code

List API to support pagination and returning list of objects

find on bcdb table raise error when searching using index

In [2]: cl = j.data.bcdb.get('zrobot')
c
In [3]: cl.tables_get('/opt/code/github/zero-os/0-robot/zerorobot/template')
Out[3]: {'service': <JumpScale9RecordChain.data.bcdb.BCDBTable.BCDBTable at 0x7f8b79ea15c0>}

In [4]: ts = cl.tables['service']

In [5]: ts.find(name='*')
Out[5]: 
[{
  "guid": "9c623dc1-46ce-49bf-9eca-54a3155141d0",
  "id": 3,
  "name": "test",
  "public": false,
  "template": "githu.com/zero-os/0-templates/nodes/0.0.1",
  "version": "0.0.1"
 }, {
  "guid": "8440cbdc-a854-42a2-be8f-4c4c66560d54",
  "id": 4,
  "name": "hello",
  "public": false,
  "template": "githu.com/zero-os/0-templates/nodes/0.0.1",
  "version": "0.0.1"
 }]

In [6]: ts.find(template='githu.com/zero-os/0-templates/nodes/0.0.1')
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
/usr/local/bin/js9 in <module>()
----> 1 ts.find(template='githu.com/zero-os/0-templates/nodes/0.0.1')

/opt/code/github/rivine/recordchain/JumpScale9RecordChain/data/bcdb/BCDBTable.py in find(self, hook, **args)
    193                 id = self.index.hget(indexkey, val)
    194                 if id is not None:
--> 195                     id = struct.unpack("<I", id)[0]
    196                     if id not in res2:
    197                         res2.append(id)

error: unpack requires a bytes object of length 4

need clear docs for orderbook

Missing docs :

• installation of gedis2 server .
• link docs with the other requirement installation like 0-db
• make the usage of https://currencylayer.com/ clear , how and why we use it

gedis needs ssl

implement client/server over ssl in gedis
test with redis over ssl (as client)

check on currency layer then use other api?

Need to make transactions clear

It's impossible in the current design to track your transaction and matching happen or not as it shown as the following there is no Ids or anything define this orders belongs to who

}
TYPE     AMOUNT  PRICE
SELL     0       10.0 XRP
SELL     100.0   10.0 XRP
BUY      50.0    10.0 ETH
BUY      100.0   10.0 ETH
BUY      100.0   10.0 ETH
BUY      100.0   10.0 ETH
BUY      100.0   10.0 ETH
BUY      100.0   10.0 ETH
BUY      100.0   10.0 ETH

Sell from multiple buy order return error

Steps

• Create three similar buy order as shown everyone it's mount=100:

x.order_book.add_buy_order(price_max='10 ETH', currency_to_buy='BTC', currency_mine=['USD','EGP'], amount=100, expiration=j.data.time.epoch + 1000, approved=True)
x.order_book.add_buy_order(price_max='10 ETH', currency_to_buy='BTC', currency_mine=['USD'], amount=100, expiration=j.data.time.epoch + 1000, approved=True)
x.order_book.add_buy_order(price_max='10 ETH', currency_to_buy='BTC', currency_mine=['USD'], amount=100, expiration=j.data.time.epoch + 1000, approved=True)

• Create sell order accept same currency as buy orders with mount= 200 , as shown
  """
  x.order_book.add_sell_order(price_min='10 XRP', currency_to_sell='BTC' ,currency_accept=['USD'], amount=200, expiration=j.data.time.epoch + 1000, approved=True)
  """

Issue

• it result the following error :

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/opt/code/github/rivine/recordchain/JumpScale9RecordChain/apps/orderbook/lib/trader.py", line 101, in _consume
    self.process(transaction)
  File "/opt/code/github/rivine/recordchain/JumpScale9RecordChain/apps/orderbook/lib/trader.py", line 90, in process
    self._post_success(transaction)
  File "/opt/code/github/rivine/recordchain/JumpScale9RecordChain/apps/orderbook/lib/trader.py", line 56, in _post_success
    if self.matcher.approved_sell_orders[sell_index]['amount'] == 0:
TypeError: list indices must be integers or slices, not NoneType
2018-07-02T16:34:50Z <Greenlet "Greenlet-0" at 0x7f6a28988e48: <bound method Trader._consume of <orderbook.lib.trader.Trader object at 0x7f6a28911390>>> failed with TypeError

• the buy order don't work corectly as shown

TYPE     id      AMOUNT  PRICE
SELL     1       0       10.0 XRP
BUY      3       0       10.0 ETH
BUY      2       0       10.0 ETH
BUY      1       100.0   10.0 ETH

Generate python client for orderbook

create any order , print wrong schema

Issue

create any order , will print schema with empty parameters like the following example

x.order_book.add_buy_order(price_max='10 ETH', currency_to_buy='BTC', currency_mine=['USD'], amount=100, expiration=j.data.time.epoch + 1000, approved=True, secret="1")

```@0xf34070e169949ac2;

struct Schema {

    comment @0 :Text;
    currencyToBuy @1 :Text;
    priceMax @2 :Data;
    amount @3 :Float64;
    expiration @4 :UInt32;
    secret @5 :Text;
    approved @6 :Bool;

    currencyMine @7 :List(Text);
    buyFrom @8 :List(Text);

Sequential ID generation for orders using gevent

finish orm layer, with capnp

Create web client for gedis

change List orders and transactions to return ids

raftos example: need to understand

check/fix/test:

• https://github.com/rivine/recordchain/tree/master/JumpScale9RecordChain/servers/raftos

check:

• https://github.com/rivine/recordchain/tree/master/JumpScale9RecordChain/servers/raft

need to understand raft & test 2 implementations to the fully

this will become a super important part of our recordchain, it needs to become our consensus engine.

also need a test where we have the gedis server integrated with the raft backend, so we can send gedis commands which get RAFT enabled (-:

0-db as backend for gedis

Implement SET, GET and INCR using 0-db as backend.

bcdb: only created required namespace in zdb

At the moment it seems that every object define in a schema result in a table in bcd and in a namespace in zdb.

But in my case, I define some objects cause I use them inside another object. like so :

@url = zrobot.task
@name = task
guid        = ''    (S)
action_name = ''    (S)
args        = {}    (dict)
state       = ''    (S)
eco         = ''    (S)
created     =       (D)

@url = zrobot.service.info
@name = info
template    = ''    (S)
version     = ''    (S)
name        = ''    (S)
guid        = ''    (S)
public      = false (B)

@url = zrobot.service
@name = service
info    =      (O) !zrobot.service.info
states  = {}   (dict)
data    = {}   (dict)
tasks   =      (LO) !zrobot.task

So here, the only object that I'm going to ever save is zrobot.service. Task and info will never be saved alone.
Is there a way to specify that task and info object should not result in a table and are just sub object of the service object ?

gedis: proper installer

currently it seems that we need to read the readme to know what are the dependencies and how to install

Can we make a simple install script or at least have a requirements.txt so we can just pip it ?

get gedis to work with tests and benchmarks

Get gedis to work with initial tests and benchmarks

bcdb: delete object

As far as I see, at the moment there is no way to delete object from the bcdb

BCDB module need to use a ZDB client not a ZDB server

Currently the bcdb module relies on the existence of a zdb server instance to work:

recordchain/JumpScale9RecordChain/data/bcdb/BCDBFactory.py

Lines 20 to 27 in 5f8f159

	def get(self, instance):
	s = j.servers.zdb.get(instance=instance)
	data = {}
	data["zdb_adminsecret_"] = s.config.data["adminsecret_"]
	data["zdb_port"] = str(s.config.data["port"])
	bcdb = self._child_class(instance=instance, data=data, parent=self, interactive=False)
	bcdb.server = s
	return bcdb

Then in the bcdb code, the zdb server instance is used to get a client to zdb:

recordchain/JumpScale9RecordChain/data/bcdb/BCDBTable.py

Line 35 in 5f8f159

self.db = self.bcdb.server.client_get(namespace=self.name, secret=s.config.data["adminsecret_"])

Which mean we actually don't need to pass by a zdb server instance at all. The current implementation also disallow to use a remote zdb since the zdb server instance has only a notion of port and not of address. So it is silently suppsed tha zdb is running locally.

So to fix this, I'll change bcdb module to requires a zdb client.

Implement orderbook (buying orders)

Generate Javascript client for orderbook

• Same logic as the python client but in Javascript

Implement orderbook (selling orders)

support ids in schemas

Currently Schema doesn't support id field
it makes it annoying to update a record in the api because API schema in oobject, will not have id field to determine which object to update, and we've to send id as separate firld which is nnoying

create Javascript client for gedis2

name of commands to be executed on gevent server should be same as their original name

Integration with atomic swap for transactions

Create a SAL layer where API for orderbook

Don, API now can talk to SAL

Add support for websockets in Gedis Server

Expired order should be removed from matching table or get expired label

When order expired, you can't know that as you will find it on the matching table but transactions will not happen on it

gedis2 readme is unclear

Some Examples :
A framework that allows for creating applications : I don't think it generates applications and the phrase itself is wrong too.
A Gedis server uses BCDB DB that saysm : ?
This is the Model layer: What other layers are there then?

js9 'j.servers.zdb.test()' broken

Document Orderbook specs and todo's

Currently it is unknown what needs to be done for the orderbook. Re-watch the recorded sessions with Kristof and write the specs in Orderbook/specs.
Also create issues for the todo's

list fields are not supported by code generation from client side