Coder Social home page Coder Social logo

ocore's Introduction

Obyte core library (Ocore)

This is a library used in Obyte clients. Some of the clients that require the library:

  • GUI wallet - GUI wallet for Mac, Windows, Linux, iOS, and Android.
  • Headless wallet - headless wallet, primarily for server side use.
  • Obyte Relay - relay node for Obyte network. It doesn't hold any private keys.
  • Obyte Hub - hub for Obyte network. Includes the relay, plus can store and forward end-to-end encrypted messages among devices on the Obyte network.

Developer guides

See the Developer resources site. Also, you'll find loads of examples in other Obyte repositories. For internal APIs, see the exports of node.js modules.

This repo is normally used as a library and not installed on its own, but if you are contributing to this project then fork, git pull, npm install, and npm test to run the tests.

Configuring

The default settings are in the library's conf.js, they can be overridden in your project root's conf.js (see the clients above as examples), then in conf.json in the app data folder. The app data folder is:

  • macOS: ~/Library/Application Support/<appname>
  • Linux: ~/.config/<appname>
  • Windows: %LOCALAPPDATA%\<appname>

<appname> is name in your package.json.

Settings

This is the list of some of the settings that the library understands (your app can add more settings that only your app understands):

conf.port

The port to listen on. If you don't want to accept incoming connections at all, set port to null, which is the default. If you do want to listen, you will usually have a proxy, such as nginx, accept websocket connections on standard port 443 and forward them to your Obyte daemon that listens on port 6611 on the local interface.

conf.storage

Storage backend -- mysql or sqlite, the default is sqlite. If sqlite, the database files are stored in the app data folder. If mysql, you need to also initialize the database with SQL file and set connection params, e.g. in conf.json in the app data folder:

{
	"port": 6611,
	"storage": "mysql",
	"database": {
		"max_connections": 30,
		"host"     : "localhost",
		"user"     : "obyte_user",
		"password" : "yourmysqlpassword",
		"name"     : "obyte_db"
	}
}

conf.bLight

Work as light client (true) or full node (false). The default is full client.

conf.bServeAsHub

Whether to serve as hub on the Obyte network (store and forward e2e-encrypted messages for devices that connect to your hub). The default is false.

conf.myUrl

If your node accepts incoming connections, this is its URL. The node will share this URL with all its outgoing peers so that they can reconnect in any direction in the future. By default the node doesn't share its URL even if it accepts connections.

conf.bWantNewPeers

Whether your node wants to learn about new peers from its current peers (true, the default) or not (false). Set it to false to run your node in stealth mode so that only trusted peers can see its IP address (e.g. if you have online wallets on your server and don't want potential attackers to learn its IP).

conf.socksHost, conf.socksPort, and conf.socksLocalDNS

Settings for connecting through optional SOCKS5 proxy. Use them to connect through TOR and hide your IP address from peers even when making outgoing connections. This is useful and highly recommended when you are running an online wallet on your server and want to make it harder for potential attackers to learn the IP address of the target to attack. Set socksLocalDNS to false to route DNS queries through TOR as well.

conf.httpsProxy

Setting for connecting through an optional HTTPS proxy. Use it when your local network can only access the Internet via an http proxy server. When both socks5 and http proxy are set, socks5 takes precedence. The configuration value is the full URL to the proxy server, eg. http://proxy:3128

conf.smtpTransport, conf.smtpRelay, conf.smtpPort, conf.smtpUser, and conf.smtpPassword

Settings for sending email. They are used e.g. if your node needs to send notifications. smtpTransport can take one of three values:

  • local: send email using locally installed sendmail. Normally, sendmail is not installed by default and when installed, it needs to be properly configured to actually send emails. If you choose this option, no other conf settings are required for email. This is the default option.
  • direct: send email by connecting directly to the recipient's SMTP server. This option is not recommended.
  • relay: send email through a relay server, like most email apps do. You need to also configure the server's host smtpRelay, its port smtpPort if it differs from the default port 25, and smtpUser and smtpPassword for authentication to the server.

MySQL conf for faster syncing

To lower disk load and increase sync speed, you can optionally disable flushing to disk every transaction, instead doing it once a second. This can be done by setting innodb_flush_log_at_trx_commit=0 in your MySQL server config file (my.ini)

Accepting incoming connections

Obyte network works over secure WebSocket protocol wss://. To accept incoming connections, you'll need a valid TLS certificate (you can get a free one from letsencrypt.org) and a domain name (you can get a free domain from Freenom). Then you accept connections on standard port 443 and proxy them to your locally running Obyte daemon.

This is an example configuration for nginx to accept websocket connections at wss://byteball.one/bb and forward them to locally running daemon that listens on port 6611:

If your server doesn't support IPv6, comment or delete the two lines containing [::] or nginx won't start

server {
	listen 80 default_server;
	listen [::]:80 default_server;
	listen 443 ssl;
	listen [::]:443 ssl;
	ssl_certificate "/etc/letsencrypt/live/byteball.one/fullchain.pem";
	ssl_certificate_key "/etc/letsencrypt/live/byteball.one/privkey.pem";

	if ($host != "byteball.one") {
		rewrite ^(.*)$ https://byteball.one$1 permanent;
	}
	if ($https != "on") {
		rewrite ^(.*)$ https://byteball.one$1 permanent;
	}

	location = /bb {
		proxy_pass http://localhost:6611;
		proxy_http_version 1.1;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
	}

	root /var/www/html;
	server_name _;
}

By default Node limits itself to 1.76GB the RAM it uses. If you accept incoming connections, you will likely reach this limit and get this error after some time:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
...
...
12: 0x3c0f805c7567
Out of memory

To prevent this, increase the RAM limit by adding --max_old_space_size=<size> to the launch command where size is the amount in MB you want to allocate.

For example --max-old-space-size=4096, if your server has at least 4GB available.

Donations

We accept donations through Kivach and forward a portion of the donations to other open-source projects that made Obyte possible.

Kivach

ocore's People

Contributors

1hyena avatar bonustrack avatar drsensor avatar fitzn avatar kakysha avatar papabyte avatar pmiklos avatar portabella avatar punqtured avatar saddam213 avatar tarmo888 avatar taump avatar thedavidmeister avatar tonyofbyteball avatar uluaiv avatar vakar8888 avatar valyakin avatar xjenekx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ocore's Issues

Serious bugs in validation.js

I found 5 calls can not be executed by the given callback function name, since I check code by running command ‘Inspect Code’ on WebStorm for file validation.js.
So, are those issues really bugs?
Thanks and look forward to you reply.

Details:

in function validateWitnesses(conn, objUnit, objValidationState, callback){ … }

-> at line 582, function ‘cb’ could not be found in this valid scope:
if (!chash.isChashValid(curr_witness))
return cb("witness address "+curr_witness+" is invalid");

in function validateMessage(conn, objMessage, message_index, objUnit, objValidationState, callback) { … }

-> at line 981, function ‘cb’ could not be found in this valid scope::
if ("address" in objSpendProof)
return cb("when single-authored, must not put address in spend proof");

-> at line 986, function ‘cb’ could not be found in this valid scope::
if (typeof objSpendProof.address !== "string")
return cb("when multi-authored, must put address in spend_proofs");

-> at line 988, function ‘cb’ could not be found in this valid scope::
if (arrAuthorAddresses.indexOf(objSpendProof.address) === -1)
return cb("spend proof address "+objSpendProof.address+" is not an author");

-> at line 993, function ‘cb’ could not be found in this valid scope::
if (objValidationState.arrInputKeys.indexOf(objSpendProof.spend_proof) >= 0)
return cb("spend proof "+objSpendProof.spend_proof+" already used");

Implement method "get_attestations" on WS API?

Would you consider adding a method on the node WS API to request attestations array for a specific address(es)?

Why? I'm working on a project that require attestations data (byteball.co) i'm already storing data on an external database but this give a single point of failure. I would like to create a software that can be used by 3p apps or individual where attestations data would be required. I want the software to not rely on a centralized db, i wish there is a method on the official node to query attestations.

"pluggable" data structure/object/interface/abstraction for DAG/unit (especially re: SQL/storage)

Taking #42 a few steps further...

This repo is intended to be a library for clients to use, but all the logic in the library is tightly coupled to SQL.

It's hard (for me) to see (upon casual inspection of the code) what code is implementing the ByteBall white paper so clients don't have to re-invent (and possibly screw up) key algorithms, and what code is just implementing SQL queries.

This causes a few issues:

  • Every client must implement SQL
  • It is harder for individual parts of the library to be split up and used independently for novel use-cases other than "wallet"
  • Implementing key logic in languages/domains other than JavaScript becomes harder
  • Writing unit tests for BB logic is much harder or impossible
  • There is no obvious abstraction/interface/data structure for DAG/unit to optimise for performance/security/legibility without auditing/refactoring the whole library for every change
  • Individual functions are just generally more complicated because they are forced to manage SQL on top of whatever other logic they need to handle

I think that if a goal was set to simply make the SQL storage backend pluggable and quarantine all SQL queries out of the "white paper logic", maybe even going as far as making the SQL storage backend a separate repo that is simply used "by default" here, then that would already help things a lot.

Implement method "get_utxos" on WS API?

Would you consider adding a method "get_utxos" on the node WebSocket API? Actually there is no practicable solution for a light wallet with more than 1000 history items to be recovered or compute utxos because of the "get_history" limitation here: https://github.com/byteball/byteballcore/blob/619f0ec29acf54abe1e8ca508dd1bcb19c772f15/light.js#L217-L218
It would be great to have a method "get_utxos" implemented on Byteball nodes or hubs so light client could get unspent outputs for any given address and broadcast tx / spend utxo. This will remove also highly reduce complexity on light client side.

A similar method was implemented in BitCoin see: bitcoin/bitcoin#4351

split out pure validation functions and SQL interactions

Problem:

  • The validation of units is a slow process because it thrashes the SQL database, causing problems syncing just a few GB of DAG (takes multiple days, some users even reported in chat syncing/validation running slower than new units are being added to the DAG)
  • There are no unit tests for the validation algorithms
  • Refactoring/using/reasoning about validation functions from the library is difficult due to high cyclomatic complexity and generally poor separation of concerns in validation.js
  • Validation is tightly coupled to specific SQL queries, forcing all clients to implement SQL if they want to validate units

Potential solution:

One of the key strengths of the design of the DAG as per the whitepaper is that both units and the DAG itself is an immutable data structure and the validity of the whole DAG and any individual unit is able to be independently and deterministically validated using an algorithm that "walks" some or all of the DAG.

This implies to me that all of the validation functions should accept data structures/objects representing DAGs and units as arguments and walk over these with no knowledge of concepts such as "SQL query" or "database connection".

Currently the validation functions like validateHashTree take a database connection and "validation state" as arguments and so are required to fetch, build and validate a hash tree ad-hoc as well as some kind of state management, rather than simply validate a hash tree.

If a DAG/unit object needs to (lazily) lookup something in the database, then it can either do it itself or an internal API that fetches/builds DAGs/units can do that. Either way, SQL isn't the responsibility of validation logic.

This approach would allow for the following:

  • DAG/unit objects, once validated, represent immutable objects and so can be cached infinitely, no need to thrash or even interact with the db once they are in memory. For the case of new units arriving from the network, we have them (and probably their parents if cached) in memory before they even hit SQL, so we should be able to validate many new units with zero SQL reads.
  • one step closer to allowing clients of the library to use storage other than SQL
  • validation can be unit tested by mocking out DAG/unit objects rather than relying on a live db connection to test anything
  • performance improvements such as implementing processing queues, parallelising/batching validation across multiple CPUs, etc. would be much easier to implement
  • consumers of the library could leverage the validation logic without a storage backend at all, as long as they implement the right data structure/object interface

ParentFingerPrint argument is not a buffer

finished headless conf
merged app root conf from /home/wilknot/headless-obyte/conf.js
app dir /home/wilknot/headless-obyte
merged user conf from /home/wilknot/.config/headless-obyte/conf.json
app dir /home/wilknot/headless-obyte
path=/home/wilknot/.config/headless-obyte/
createDatabaseIfNecessary byteball.sqlite
takeConnectionFromPool will wait for ready
will archive if exists from unit N6QadI9yg3zLxPMphfNGJcPfddW4yHPkoGMbbGZsWa0=
takeConnectionFromPool will wait for ready
takeConnectionFromPool will wait for ready
stat null
db is now ready
opening new db connection
db is now ready
db is now ready
opened db
db version 22, software version 22

remote access allowed from devices: DEVICE ALLOWED TO CHAT
payouts allowed to address: WHERE THE MONEY CAN BE SENT TO

Passphrase: moqueued jobs: [], locked keys: []
starting network
initCaches
lock acquired [ 'dependencies' ]
setDevicePrivateKey
same device addresses: 0BCK5JVCY3G6ONL3ITWLXE445XBK62OYN
objMyTempDeviceKey not set yet, can't log in
assert.js:350
throw err;
^

AssertionError [ERR_ASSERTION]: parentFingerPrint argument is not a buffer, it's undefined
at checkBuffer (/home/wilknot/headless-obyte/node_modules/bitcore-lib/lib/hdpublickey.js:339:5)
at Function.HDPublicKey._validateBufferArguments (/home/wilknot/headless-obyte/node_modules/bitcore-lib/lib/hdpublickey.js:347:3)
at HDPublicKey._buildFromBuffers (/home/wilknot/headless-obyte/node_modules/bitcore-lib/lib/hdpublickey.js:296:15)
at HDPublicKey._buildFromObject (/home/wilknot/headless-obyte/node_modules/bitcore-lib/lib/hdpublickey.js:257:15)
at new HDPublicKey (/home/wilknot/headless-obyte/node_modules/bitcore-lib/lib/hdpublickey.js:59:23)
at Object.HDPublicKey (/home/wilknot/headless-obyte/node_modules/bitcore-lib/lib/hdpublickey.js:39:12)
at createWallet (/home/wilknot/headless-obyte/start.js:156:27)
at /home/wilknot/headless-obyte/start.js:95:7
at /home/wilknot/headless-obyte/start.js:220:3
at /home/wilknot/headless-obyte/node_modules/ocore/sqlite_pool.js:240:5

network.js failed to reroute command to the next peer: Cannot read property 'responseHandlers' of undefined

My headless-byteball wallet died with the error below (on Jan 31). The byteballcore version was 0.1.35

/home/pi/headless-byteball/node_modules/byteballcore/network.js:171
				ws.assocPendingRequests[tag].responseHandlers.forEach(function(rh){
				                            ^

TypeError: Cannot read property 'responseHandlers' of undefined
    at /home/pi/headless-byteball/node_modules/byteballcore/network.js:171:33
    at /home/pi/headless-byteball/node_modules/byteballcore/network.js:251:11
    at tryFindNextPeer (/home/pi/headless-byteball/node_modules/byteballcore/network.js:266:3)
    at findNextPeer (/home/pi/headless-byteball/node_modules/byteballcore/network.js:249:2)
    at EventEmitter.<anonymous> (/home/pi/headless-byteball/node_modules/byteballcore/network.js:255:4)
    at EventEmitter.g (events.js:291:16)
    at emitOne (events.js:101:20)
    at EventEmitter.emit (events.js:188:7)
    at /home/pi/headless-byteball/node_modules/byteballcore/network.js:577:13
    at /home/pi/headless-byteball/node_modules/byteballcore/network.js:203:3

Proposal: implement a method in WS API to get address definition

First time a newly created address post an unit on the DAG it required the address definition inside the unit authors field. We are working on a library that need to know in advance when posting an unit if the author address has already posted the definition or not on the DAG. Actually we using the method light/get_history to see if the address has already a definition. The problem is that this method can give huge results because it come with all user unit history, it may also not working for user with an history bigger than 2000 items due to this limitation: https://github.com/byteball/byteballcore/blob/9dfba8cf485b2da3251e7d683d2b203cf80962fe/light.js#L19 (even if we see limit error we can assume user already posted definition it's not 100% garantee cuz there could be only incoming unit history)

Would it make sense to have a method implement on the WS API that could be get_definition to have an easy way to check if user need or not to post his definition on the DAG? It may also be used to read smart contract definition. Or is there already a better solution?

Some errors when initializing mysql

Tried this on fresh installation on Ubuntu 16.04

For table pairing_secrets: "Invalid default value for 'expiry_date'"
Solution: change "expiry_date TIMESTAMP NOT NULL" to "expiry_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP"

For table wallet_signing_paths: "All parts of a PRIMARY KEY must be NOT NULL; if you need NULL in a key, use UNIQUE instead"
Solution: change "PRIMARY KEY (wallet, signing_path)," to "UNIQUE KEY (wallet, signing_path),"

For table shared_address_signing_paths: "All parts of a PRIMARY KEY must be NOT NULL; if you need NULL in a key, use UNIQUE instead"
Solution: change "PRIMARY KEY (shared_address, signing_path)" to "UNIQUE KEY (shared_address, signing_path)"

Hope this helps.

Proposal: implement a method in WS API to get address balances

In the current WS API there is no easy way to get an account balances. There is the method light/get_history which give in and out transactions of a specific address(es) but it's not that easy to compute the balances giving the possibilities that there could be smart contract, asset created or it could simply not work if there is more transactions than the MAX_HISTORY_ITEMS limit. It would be great to have directly the balance from a new API method without having to compute it yourself from history.

slack integration with #tech?

It might be nice to see notifications from key byteball repos, including this one, coming through the #tech channel in slack

Proposal: implement a method in WS API to get someone profile

Actually anyone can post their public profile from the wallet or byteball.co using the message "profile". It would be great if we could also get the complete user profile directly from the WS API. Now there no easy way for a light wallet to get someone complete profile.

Ideally the method should make sense of the complete profile and allow user to update or delete any field(s). In a way that when an user post this profile payload:

{
  name: 'Fabien',
  location: 'Thailand',
  email: '[email protected]'
}

then later post:

{
  location: 'France',
  email: null,
  age: 30
}

With get_profile we would get this result:

{
  name: 'Fabien',
  location: 'France',
  age: 30
}

With this change Byteball 3p apps could leverage this data to enable social features.

getSourceString is not uniquely decodable

(Feel free to resolve this as Won't Fix based on the limited set of potential strings that can be keys in a Unit dictionary. This is merely proof-of-concept for something that struck me as weird.)

The object_hash module uses the getSourceString function to turn a Unit into a String before hashing the String to produce a content_hash (among other things). The getSourceString function produces the same output String for different inputs, which, in turn, produces an identical hash. This is because the object type is not tokenized, and this could theoretically cause an issue as it breaks the widespread assumption that different inputs of the same data type (e.g., Unit) will hash to different values.

This example input:

var objectOne = ['s', 'a'];
var sourceStringOne = getSourceString(objectOne);

var objectTwo = [{s: {s: 'a'}}];
var sourceStringTwo = getSourceString(objectTwo);

console.log("Object One: " + sourceStringOne);
console.log("Object Two: " + sourceStringTwo);
console.log("Strings match? " + (sourceStringOne === sourceStringTwo));

produces this output (including unprintable characters):

Object One: [�s�s�s�a�]
Object Two: [�s�s�s�a�]
Strings match? true

Again, this case is extremely unlikely (maybe impossible) in this code base since the Unit object's keys are fixed, but I figured I'd call it out.

"Error: not 1 unit" when checking witnessed level

The following error is thrown when the witnessed level of the genesis unit is checked.

checkWitnessedLevelAndReplace 3gLI9EnI2xe3WJVPwRg8s4CB24ruetuddS0wYa2EI3c=
/root/node_modules/byteballcore/storage.js:1297
			throw Error("not 1 unit");
			^

Error: not 1 unit
    at /root/node_modules/byteballcore/storage.js:1297:10
    at Statement.<anonymous> (/root/node_modules/byteballcore/sqlite_pool.js:113:6)

It looks like this recent commit introduced it. See more details in the comment I added there.

readUnitProps fails with 'different props' on equal units

Using mysql database, seemingly equal units are reported as different when comparing cached and db versions:

unit null, best parent 3BoTBCE4yfFKL1kmyCJmO0d1E8xKA9Qom3MKdu4HveA=, wlevel 29444
node_modules/mysql/lib/protocol/Parser.js:80
        throw err; // Rethrow non-MySQL errors
        ^

Error: different props of 3BoTBCE4yfFKL1kmyCJmO0d1E8xKA9Qom3MKdu4HveA=, mem: {"unit":"3BoTBCE4yfFKL1kmyCJmO0d1E8xKA9Qom3MKdu4HveA=","level":29445,"latest_included_mc_index":null,"main_chain_index":null,"is_on_main_chain":0,"is_free":1,"is_stable":0,"witnessed_level":29444}, db: {"unit":"3BoTBCE4yfFKL1kmyCJmO0d1E8xKA9Qom3MKdu4HveA=","level":29445,"latest_included_mc_index":null,"main_chain_index":null,"is_on_main_chain":0,"is_free":1,"is_stable":0,"witnessed_level":29444}
    at node_modules/byteballcore/storage.js:714:12
    at Query._callback (node_modules/byteballcore/mysql_pool.js:43:4)
    at Query.Sequence.end (node_modules/mysql/lib/protocol/sequences/Sequence.js:88:24)
    at Query._handleFinalResultPacket (node_modules/mysql/lib/protocol/sequences/Query.js:139:8)
    at Query.EofPacket (node_modules/mysql/lib/protocol/sequences/Query.js:123:8)
    at Protocol._parsePacket (node_modules/mysql/lib/protocol/Protocol.js:279:23)
    at Parser.write (node_modules/mysql/lib/protocol/Parser.js:76:12)
    at Protocol.write (node_modules/mysql/lib/protocol/Protocol.js:39:16)
    at Socket.<anonymous> (node_modules/mysql/lib/Connection.js:103:28)
    at emitOne (events.js:116:13)

Missing binary keyword in byteball.sql

Hi,

The byteball.sql file on the master and testnet branches is currently missing a binary keyword for the private_profiles table. The result is that the table can't be created. It should be:

unit CHAR(44) BINARY NOT NULL

Regards,

Matthijs

mysql: ERROR 1215 (HY000) at line 677: Cannot add foreign key constraint

When creating the MySQL database using byteball.sql the creation of the asset_metadata table fails because the asset column specification is not compatible with the unit column in the assets table used as foreign key reference. Probably the same for the metadata_unit foreign key.
Adding BINARY to the column spec and ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; to the table specification would solve it.

ERROR 1215 (HY000) at line 677: Cannot add foreign key constraint

resend multi-sig wallet creation notification

Is there a way to resend a multi-sig wallet creation notice to a device? I have this weird situation where I wanted to create a 2-of-3 wallet. On the 2nd device, I got the pop-up asking if I wanted to create the wallet but on the 3rd device, I never got the notification. The creation process is now stuck since I can't see any trace of the wallet on the third device and when I select the wallet under creation on the two other devices, the wallet crashes. Same behavior on Android and Windows wallet (device 1 and 2)

Question : what's the purpose of witness proof in catch up process ?

Hi, Tony & other contributors

  • I started reading byteball code about one month ago , I know that one fresh client need to do catch up to pull old data from peers and each wallet may have different witness list . at the beginning of catch up, wallet need to send its witness list and last know mci to destination peer, then the peer got it and start to calculate the 'last_ball_mci' and 'last_ball_unit ' which is designed to indicate where the catch up ends.

  • What I confused is why the 'last_ball_mci' depends on clients' witnesses since the catch up data were historical and old. what happen if there is no enough units associated with passed witnesses ?
    will the catch up request be rejected ? why ?

    I am new to byteball ,appreciate for your explanation ! @tonyofbyteball @kakysha

issue when syncing chain data

When I ran headless-byteball, the error was thrown and process terminates

From this point, output will be redirected to /root/.config/headless-byteball/log.txt
To release the terminal, type Ctrl-Z, then 'bg'
/root/headless-byteball/node_modules/byteballcore/storage.js:730
                                        throw Error("different props of "+unit+", mem: "+JSON.stringify(props2)+", db: "+JSON.stringify(props));
                                        ^

Error: different props of 80ftUd7v34yJstagU5EwTu6wyyDW0yRk9XX/K8ZL36Y=, mem: {"unit":"80ftUd7v34yJstagU5EwTu6wyyDW0yRk9XX/K8ZL36Y=","level":3350626,"latest_included_mc_index":3245724,"main_chain_index":3245725,"is_on_main_chain":1,"is_free":1,"is_stable":0,"witnessed_level":3350611,"headers_commission":344,"payload_commission":157,"sequence":"good","author_addresses":["CHNK74ZY66XTYLWB45UES26G3AAS522V"],"witness_list_unit":"oj8yEksX9Ubq7lLc+p6F2uyHUuynugeVq4+ikT67X6E="}, db: {"unit":"80ftUd7v34yJstagU5EwTu6wyyDW0yRk9XX/K8ZL36Y=","level":3350626,"latest_included_mc_index":3245724,"main_chain_index":3245725,"is_on_main_chain":1,"is_free":1,"is_stable":0,"witnessed_level":3350611,"headers_commission":344,"payload_commission":157,"sequence":"temp-bad","author_addresses":["CHNK74ZY66XTYLWB45UES26G3AAS522V"],"witness_list_unit":"oj8yEksX9Ubq7lLc+p6F2uyHUuynugeVq4+ikT67X6E="}
    at /root/headless-byteball/node_modules/byteballcore/storage.js:730:12
    at Statement.<anonymous> (/root/headless-byteball/node_modules/byteballcore/sqlite_pool.js:115:6)

seems there is a conflict, can any one help me solve this problem please.

Find min witnessed level has problem when witness list of free main chain unit and last stable unit are different

Below is code in "updateStableMcFlag" function of main_chain.js to find min witnessed level.
If free main chain unit and last stable unit has different witness list, for example, there is one witness different. From free main chain unit to unit whose level large than witnessed_level of free main chain unit along main chain, there may be only 6 witnesses are in witness list of last stable unit.

conn.query("SELECT witnessed_level FROM units WHERE is_free=1 AND is_on_main_chain=1", function(wl_rows)
{
    if (wl_rows.length !== 1)
        throw Error("not a single mc wl");
    // this is the level when we colect 7 witnesses if walking up the MC from its end
    var mc_end_witnessed_level = wl_rows[0].witnessed_level;
    conn.query(
    // among these 7 witnesses, find min wl
    "SELECT MIN(witnessed_level) AS min_mc_wl FROM units LEFT JOIN unit_authors USING(unit) \n\
    WHERE is_on_main_chain=1 AND level>=? AND address IN(?)", // _left_ join enforces the best query plan in sqlite
    [mc_end_witnessed_level, arrWitnesses],
    function(min_wl_rows){
    if (min_wl_rows.length !== 1)
        throw Error("not a single min mc wl");
    var min_mc_wl = min_wl_rows[0].min_mc_wl;
...

Unnessary judgement when compose joint

In composer.js , after the code that judge the type of client

	if (conf.bLight && !params.lightProps){
		var network = require('./network.js');
		network.requestFromLightVendor(
			'light/get_parents_and_last_ball_and_witness_list_unit', 
			{witnesses: arrWitnesses}, 
			function(ws, request, response){
				if (response.error)
					return params.callbacks.ifError(response.error);
				if (!response.parent_units || !response.last_stable_mc_ball || !response.last_stable_mc_ball_unit || typeof response.last_stable_mc_ball_mci !== 'number')
					return params.callbacks.ifError("invalid parents from light vendor");
				params.lightProps = response;
				composeJoint(params);
			}
		);
		return;
	}

(line 383)

, the program judge it again.

	if (conf.bLight && !lightProps)
		throw Error("no parent props for light");

(line 436)

Why?

If I am using a light client, after I get all information from the vendor, the params.lightProps will be deleted and the lightProps must be null, and the the code will be an unnecessary code. Am I thinking right?

ERROR 1022 (23000) at line 724: Can't write; duplicate key in table 'attested_fields'

The latest byteball.sql fails to execute on Mysql database:

ERROR 1022 (23000) at line 724: Can't write; duplicate key in table 'attested_fields'

The problem is that constraint attestationsByAttestorAddress is already used in table attestations.

Also the unit field in tables attested_fields and original_addresses needs the BINARY keyword:

	unit CHAR(44) BINARY NOT NULL,

'new_my_transactions' never fired

I tried to follow the manual and make a basic chat bot, but unfortunately, i wasn't able to control incoming transactions.
I was succesfully issuing new addresses for new users, but when they send byte to this addresses, no events were fired at all.

I used fresh install Ubuntu 16.04 with sqlite database (because mysql had some errors here and there).

And there is no documentation to figure out, what is happening and what are the possible solutions.

Really sad, that such a great idea is being developed by only one person, of course it is impossible to develop, promote and carefully document such pretty complicated software.

suggestion for the protocol - make 12 the minimum number of witnesses rather than the exact number

The whitepaper explains that the choice of 12 witnesses is somewhat arbitrary.

The above means that each unit must list its witnesses so that they can be compared. We require that the number of witnesses is exactly 12. This number 12 was selected because:

it is sufficiently large to protect against the occasional failures of a few witnesses (they might prove dishonest, or be hacked, or go offline for a long time, or lose their private keys and go offline forever);

it is sufficiently small that humans can keep track of all the witnesses to know who is who and change the list when necessary;

the one allowed mutation is sufficiently small compared with the 11 unchanged witnesses.

So, the reasons for it being large are to protect the overall reliability of the network, the reasons to keep it small are to make it easier for humans to understand and come to a consensus on who the witnesses should be.

It seems that 12 is already deemed an acceptable minimum for network reliability, and can be a minimum enforced by the protocol.
Why not let the maximum be set by humans?

function pickParentUnits has no logic to ensure unit serial when number of free good sequence units large than 2 * MAX_PARENTS_PER_UNIT

function pickParentUnits only select first MAX_PARENTS_PER_UNIT free good sequence units order by unit. It seems that if number of free good sequence units larger than (2 * MAX_PARENTS_PER_UNIT), there is no logic to ensure unit serial.

For example, there is 32 free units(unit1 ... unit32), they have same parent. I send a new unit A and unit1 ...unit16 are selected as parents, then I send another unit B, it may select unit17 ... unit32 as parents, so now unit A and B are non serial units

Missing break instruction in wallet_defined_by_addresses.js

See line 345.

This bug causes an exception in case an address definition contains a 'r set of' and needs approval by multiple users because the missing break let the switch continue to evaluate possibilities until it meets its end (throw exception).

Found while trying to implement payments with shared addresses.

I fixed it locally by adding the missing 'break'. I am extensively analysing the core: if you granted me the right to branch and create pull requests I'd be glad to provide ready solutions together with bug notifications.

Best regards,

Yary

Question: Can user change a shared_address's definition by sending a definition_change unit ?

If I make a shared_address with someone, and we all can use the money in it, so we all can compose unit with this address, right? So I am thinking, if I can compose a definition_change unit to change this shared_address's definition so the other one will no longer be able to use this address.

And another question came out. Whether a shared_address's definition can be used in definition_change unit to be an another address's new definition?

Node can not be synced

Getting this error:

Error: known bad 1o1lmOCb2Fs5aa9q1BuvkqPXQTc7b+sDvMgC7QMC5f0=: too many (2) witness list mutations relative to MC unit e1luhU51xhFer3UMBoIVVQ+QfdMNRTUpjJ/6a94H9Pc=
at Object.ifKnownBad (/home/byteball/BetByteBall/node_modules/byteballcore/network.js:725:12)

I deleted the base and decided to start again... There is nothing anyone can do with database, once it gets this kind of error. I believe it should be handled properly, so the node could continue to sync.

NakedUnit calculation is not explicit about fields

In object_hash.js, the getNakedUnit function makes a deep copy of the provided objUnit and then deletes several fields that are not to be included in what constitutes a "naked unit". This runs the risk of leaving extraneous fields from objUnit that should not be included when passing the resulting "naked unit" to other functions such as, getBase64Hash.

On the one hand, it's possible to argue that getNakedUnit operates this way to be a "cleaning" function for excluding certain known fields. However, given how the code uses getNakedUnit, it's clear that it is intended to prepare the objUnit for hashing. As a result, failing to remove rogue fields will influence the unit's hash and corrupt the result.

One possible solution would be to not use a deep copy + delete, but instead construct a new object and selectively copy all fields from objUnit that should be present. This approach protects from the issue above and has the additional benefit of being explicit about what fields should be in a valid "naked unit", which makes it easier to reason about.

"inputs and outputs do not balance" - composer and validator inconsistency

After posting the below multi-payment unit, I get the following validation error:

{"error":"inputs and outputs do not balance: 1000 !== 1000 + 681 + 508"}

It looks like the payment validation doesn't know which payment message was used to pay the fees. In this example, the fees are paid from the second message, still the validation assumes it was paid from the first payment message.

Expected behavior: the composer should add the fee paying message as first if the validation assumes that; or the validation should be intelligent enough to find out which message paid the fees.

I used the divisible_asset.composeAndSaveDivisibleAssetPaymentJoint function to generate this unit with an extra payment message which the composer placed in the first position.

A calculation that proves the fees are paid from the second payment:

sqlite> select sum(amount) from outputs where unit='u0EIU2LlYc87lYrM7KukLdq5/I6zMHPatpiU192m7Xk=' and is_spent=0 and asset is null;
sum(amount) = 4959

4959 - 681 - 508 = 3770

Unit that I tried to post on the testnet and resulted in the validation the error:

{ unit: 
   { version: '1.0t',
     alt: '2',
     messages: 
      [ { app: 'payment',
          payload_location: 'inline',
          payload_hash: 'c/AXJ51Jc0gAcW5PGPPDK4LUgmlOF6pC2Y1Vvp1VPwM=',
          payload: 
           { inputs: 
              [ { unit: 'wTsnuKp7J0GrC/2FcJTg17cZgYmOEIdJshX074tYFQ8=',
                  message_index: 0,
                  output_index: 1 } ],
             outputs: 
              [ { address: 'XDS4TZUHQUH5Q2N2OON27AC6CJHDG3RP', amount: 1000 } ] } },
        { app: 'payment',
          payload_location: 'inline',
          payload_hash: 'IPOL2dD2TyqJBu5xYRCX986Nzlk12lBcqIIJloMEPSU=',
          payload: 
           { outputs: 
              [ { address: 'XDS4TZUHQUH5Q2N2OON27AC6CJHDG3RP', amount: 3770 } ],
             inputs: 
              [ { unit: 'u0EIU2LlYc87lYrM7KukLdq5/I6zMHPatpiU192m7Xk=',
                  message_index: 0,
                  output_index: 1 } ] } },
        { app: 'payment',
          payload_location: 'inline',
          payload_hash: '1alxwnrl8UKWoJnH4qlWtPLLm9lngc8wHYEQ7TZxtmQ=',
          payload: 
           { asset: 'x+otCo902jUxw+coPegITxp+3B9ldj5+1h8/ojjTjLk=',
             inputs: 
              [ { type: 'issue',
                  amount: 1,
                  serial_number: 1,
                  address: 'XDS4TZUHQUH5Q2N2OON27AC6CJHDG3RP' } ],
             outputs: [ { address: '6YFMUDBYT7254E5HKAFT4VTXHV47PSCG', amount: 1 } ] } } ],
     authors: 
      [ { address: 'HLDTGMYTLIQHWRVXD3QLA7QAX4OQPORU',
          authentifiers: 
           { 'r.1.0': 'raXSNKWwxRzPnfRd+wnQM3L36VnwKDssMuZkx9AeI+lP2gzWzc3xcwhI8AcglS/gjsZp7yU8jnguaSDTIe7RtA==' },
          definition: 
           [ 'or',
             [ [ 'address', '6YFMUDBYT7254E5HKAFT4VTXHV47PSCG' ],
               [ 'and',
                 [ [ 'address', 'XDS4TZUHQUH5Q2N2OON27AC6CJHDG3RP' ],
                   [ 'has',
                     { what: 'output',
                       asset: 'x+otCo902jUxw+coPegITxp+3B9ldj5+1h8/ojjTjLk=',
                       amount_at_least: 1,
                       address: '6YFMUDBYT7254E5HKAFT4VTXHV47PSCG' } ] ] ] ] ] },
        { address: 'XDS4TZUHQUH5Q2N2OON27AC6CJHDG3RP',
          authentifiers: 
           { r: 'raXSNKWwxRzPnfRd+wnQM3L36VnwKDssMuZkx9AeI+lP2gzWzc3xcwhI8AcglS/gjsZp7yU8jnguaSDTIe7RtA==' } } ],
     earned_headers_commission_recipients: 
      [ { address: 'XDS4TZUHQUH5Q2N2OON27AC6CJHDG3RP',
          earned_headers_commission_share: 100 } ],
     parent_units: [ 'n0lDwsjGdmyHPOGKLaE02JYQ/1ltSQh45X46ev4WHxg=' ],
     last_ball: '7N+9RzKVq9YPVPB8duFKxMPrKhYP9GOA7PQuIyznnKA=',
     last_ball_unit: 'G1At3W8Coee+5dgqgK4BoCBHwwFWC2M47ouWv+mymdA=',
     witness_list_unit: 'TvqutGPz3T4Cs6oiChxFlclY92M2MvCvfXR5/FETato=',
     headers_commission: 681,
     payload_commission: 508,
     unit: '7xzYjADPxvRJBSkqNERzIQtYbGA79Ac0wNO6ZTdrzgM=' } }

Catchup is slow when tps is more than 60

Lately, I build a test-net in my computer, and I remove the limit of witnesses, so when any new joint is composed, the witness will compose a witness joint. And after a few minutes, I start a new full-wallet and try to catch up the chain. A interesting thing happened: the amount of both units and unhandled joints is increasing and the amount of unhandled joints seems will never get less.

Cannot read property 'connectionConfig' of undefined

The latest git version of byteballcore fails on startup when used with mysql. The issue is in function initUnstableUnit where db.escape is used as a mapper function, but the escape function refers to this which in this usage gets the wrong context:

"SELECT parent_unit, child_unit FROM parenthoods WHERE child_unit IN("+Object.keys(assocUnstableUnits).map(db.escape)+")",

the fix is converting it to a function call:

.map(function() { return db.escape(); })

The Error:

node_modules/mysql/lib/protocol/Parser.js:80
        throw err; // Rethrow non-MySQL errors
        ^

TypeError: Cannot read property 'connectionConfig' of undefined
    at Pool.escape (node_modules/mysql/lib/Pool.js:281:42)
    at Array.map (<anonymous>)
    at node_modules/byteballcore/storage.js:1248:108
    at Query._callback (node_modules/byteballcore/mysql_pool.js:43:4)
    at Query.Sequence.end (node_modules/mysql/lib/protocol/sequences/Sequence.js:88:24)
    at Query._handleFinalResultPacket (node_modules/mysql/lib/protocol/sequences/Query.js:139:8)
    at Query.EofPacket (node_modules/mysql/lib/protocol/sequences/Query.js:123:8)
    at Protocol._parsePacket (node_modules/mysql/lib/protocol/Protocol.js:279:23)
    at Parser.write (node_modules/mysql/lib/protocol/Parser.js:76:12)
    at Protocol.write (node_modules/mysql/lib/protocol/Protocol.js:39:16)

Error: Looks like you are loading multiple copies of byteballcore, which is not supported.

I'm trying to start write script for issuing new asset, following instructions provided in wiki here

I write only three lines of code and getting this error

"use strict";

var headlessWallet = require('headless-byteball');
var composer = require('byteballcore/composer.js');

I'm pretty new to npm and Node.js and maybe i'm making something completely wrong, but as I understand problem is in calling conf.js at byteballcore package by both 'require' commands, with first command setting

global._bByteballCoreLoaded = true

by

require('./enforce_singleton.js');

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.