origintrail / dkg-client Goto Github PK

On Nikola's guidance in Discord, suggest having a C# .NET implementation of the DKG client for those on the Microsoft stack.

I would like to try integrating this with a social network instance like joinmastodon.org, mainly to solve the global search and discoverability problem of the Fediverse.

1. model the db schema as a standards-compliant graph
2. add middleware in the server to CRUD entities to the graph

Is this client able to support writing 1000 datasets per second to the graph?

I assume the bottleneck is either the network latency or db engine.

I was already running the ot-node using v5 with one (1) DC node and two (2) DH nodes running.
I followed the DKG client library v5 steps here to publish using simplePrepare https://docs.origintrail.io/developers/using-dkg-v5/devsinstall#publishing-data-using-the-dkg-client-library, however it gives error Error: Failed to choose holders. Not enough DHs submitted, which based on review of the ot-node codebase appears to be because I need to have at least three (3) DH nodes running to provide sufficient verified replications.

Log of running DKG Client library v5
git clone https://github.com/OriginTrail/dkg-client cd dkg-client nvm use v9.11.2 npm install node

		```
		> var OT_NODE_HOSTNAME = "0.0.0.0";
		> var OT_NODE_PORT = "8900";
		> var DKGClient = require('/Users/cuttlefish/code/github/dkg-client/index.js');
		> DKGClient
		[Function: DKGClient]
		> var dkg = new DKGClient();
		> dkg
		DKGClient {}
		> dkg.init(OT_NODE_HOSTNAME,OT_NODE_PORT,false);
		> const simpleDataset = dkg.simplePrepare({
		...     identifier_type: 'YOUR_TYPE', // e.g. DID, SGTIN, URL etc
		...     identifier_value: '12345',
		...     properties:{
		.....         hello: 'world'
		.....     }
		... });
		> simpleDataset
		'\n{\n  "@graph": [\n    {\n      "@id": "12345",\n      "@type": "otObject",\n      "identifiers": [\n        {\n          "@type": "YOUR_TYPE",\n          "@value": "12345"\n        }\n      ],\n      "properties": {"hello":"world"},\n      "relations": []\n    }\n  ]\n}\n'
		> const simpleDataset = dkg.simplePrepare({
		...     identifier_type: 'YOUR_TYPE', // e.g. DID, SGTIN, URL etc
		...     identifier_value: '12345',
		...     properties:{
		.....         hello: 'world'
		.....     }
		... 
		> // publishing a dataset
		> dkg.publish(simpleDataset).then((publish_result)=>{
		...     console.log(publish_result);
		... });
		Sending import request.
		Promise {
		  <pending>,
		  domain: 
		   Domain {
		     domain: null,
		     _events: 
		      { removeListener: [Function: updateExceptionCapture],
		        newListener: [Function: updateExceptionCapture],
		        error: [Function: debugDomainError] },
		     _eventsCount: 3,
		     _maxListeners: undefined,
		     members: [] } }
		> Import result status: COMPLETED
		Sending replication request.
		Replication result status: PENDING
                  ...
		Replication result status: PENDING
		Replication result status: FAILED
		(node:48126) UnhandledPromiseRejectionWarning: Error: Replication failed. Reason: undefined.
		    at DKGClient._getReplicationResult (/Users/cuttlefish/code/github/dkg-client/index.js:320:19)
		    at <anonymous>
		    at process._tickCallback (internal/process/next_tick.js:182:7)
		(node:48126) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
		(node:48126) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
		```

Logs from the OT-node v5
2022-10-24T00:09:24.827Z - api - POST: Replicate request received. 2022-10-24T00:09:24.869Z - trace - [stargazer] Using default trac price in base currency from configuration: 0.00005 2022-10-24T00:09:24.870Z - trace - [stargazer] Using default gas price from configuration: 20000000000 2022-10-24T00:09:24.870Z - trace - Calculated offer price for data size: 0.002111MB, and holding time: 0.006944444444444444 days, PRICE: 120016244229744740000[mTRAC] 2022-10-24T00:09:24.877Z - trace - [stargazer] Get profile by identity 0x2fa6d32e314aab43a58999d6f5532a15418da153 2022-10-24T00:09:24.927Z - trace - [stargazer] Get minimum stake from blockchain 2022-10-24T00:09:24.958Z - info - Exporting dataset: 0x9389afa0dc8b3a177461b577162085acc740f627bfbdb70488041e823b8080d5. For internal offer id: a5848007-8fcf-42b1-ae52-ec3c27063ad5. 2022-10-24T00:09:24.977Z - info - Preparing data for offer with internal id: a5848007-8fcf-42b1-ae52-ec3c27063ad5, started. (node:48552) Warning: N-API is an experimental feature and could change at any time. 2022-10-24T00:09:37.017Z - info - Preparing data for offer with internal id: a5848007-8fcf-42b1-ae52-ec3c27063ad5, completed successfully. 2022-10-24T00:09:37.112Z - trace - [stargazer] Using default gas price from configuration: 20000000000 2022-10-24T00:09:37.113Z - trace - [stargazer] createOffer (0x2fa6d32e314aab43a58999d6f5532a15418da153, 0x9389afa0dc8b3a177461b577162085acc740f627bfbdb70488041e823b8080d5, 0x9f9c70de04049a71ef0edb819c1c588d2f938cce1d53dca174f4f9c614af24d4, 0x106d1e85c1558430faba36255b066b436ed45fbd4ddce05a803b99cec60af448, 0x81923da63de846527eb78635643dc13ae9a2c6a27b679c8636e0b96ec3cc1b9a, 0x41839f9f2ae6f4fb01c51ac3e8ac2ac1c9505d8e6e34d5964d7190555b390e3b, 0x781e64952478359bd10b79ab69d42bea189834be, 10, 120016244229744740000, 2111, 1) 2022-10-24T00:09:37.135Z - trace - Sending transaction to blockchain, nonce 60, balance is 999999999998456438980000000000 2022-10-24T00:09:37.520Z - important - Offer with internal ID a5848007-8fcf-42b1-ae52-ec3c27063ad5 for data set 0x9389afa0dc8b3a177461b577162085acc740f627bfbdb70488041e823b8080d5 written to blockchain stargazer. Waiting for DHs... 2022-10-24T00:09:57.813Z - trace - Offer successfully started for data set 0x9389afa0dc8b3a177461b577162085acc740f627bfbdb70488041e823b8080d5 on blockchain stargazer. Offer ID 0x77fad6d671c54ea552aae951edc4f3ae573f8450f1a2ac055af1a927ce86b6bb. Internal offer ID a5848007-8fcf-42b1-ae52-ec3c27063ad5. 2022-10-24T00:10:32.734Z - info - Received replication request for offer_id 0x77fad6d671c54ea552aae951edc4f3ae573f8450f1a2ac055af1a927ce86b6bb from node 85b7d412ab88d04136568e7618cfeb0ce4e9c94b. 2022-10-24T00:10:32.742Z - info - Sending replication request acknowledgement for offer_id 0x77fad6d671c54ea552aae951edc4f3ae573f8450f1a2ac055af1a927ce86b6bb to node 85b7d412ab88d04136568e7618cfeb0ce4e9c94b. 2022-10-24T00:10:37.719Z - info - Received replication request for offer_id 0x77fad6d671c54ea552aae951edc4f3ae573f8450f1a2ac055af1a927ce86b6bb from node 4ab6cd2fb8330cea9397915ef1adaa5e4290a7a3. 2022-10-24T00:10:37.728Z - info - Sending replication request acknowledgement for offer_id 0x77fad6d671c54ea552aae951edc4f3ae573f8450f1a2ac055af1a927ce86b6bb to node 4ab6cd2fb8330cea9397915ef1adaa5e4290a7a3. 2022-10-24T00:10:53.524Z - trace - Loaded replication from file for offer internal ID a5848007-8fcf-42b1-ae52-ec3c27063ad5 and color red 2022-10-24T00:10:53.757Z - info - Successfully sent replication data for offer_id 0x77fad6d671c54ea552aae951edc4f3ae573f8450f1a2ac055af1a927ce86b6bb to node 4ab6cd2fb8330cea9397915ef1adaa5e4290a7a3. 2022-10-24T00:10:54.141Z - trace - [stargazer] Reading HOLDING_CONTRACT contract version. 2022-10-24T00:10:54.188Z - notify - Replication finished for DH node 4ab6cd2fb8330cea9397915ef1adaa5e4290a7a3 2022-10-24T00:11:13.237Z - trace - Loaded replication from file for offer internal ID a5848007-8fcf-42b1-ae52-ec3c27063ad5 and color blue 2022-10-24T00:11:13.462Z - info - Successfully sent replication data for offer_id 0x77fad6d671c54ea552aae951edc4f3ae573f8450f1a2ac055af1a927ce86b6bb to node 85b7d412ab88d04136568e7618cfeb0ce4e9c94b. 2022-10-24T00:11:13.933Z - trace - [stargazer] Reading HOLDING_CONTRACT contract version. 2022-10-24T00:11:13.985Z - notify - Replication finished for DH node 85b7d412ab88d04136568e7618cfeb0ce4e9c94b 2022-10-24T00:11:27.848Z - notify - Replication window for 0x77fad6d671c54ea552aae951edc4f3ae573f8450f1a2ac055af1a927ce86b6bb is closed. Replicated to 2 peers. Verified 2. 2022-10-24T00:11:27.849Z - error - Failed to process command dcOfferChooseCommand and ID 2453bfce-6441-4a42-8439-dac913581465. Error: Failed to choose holders. Not enough DHs submitted.. Error: Failed to choose holders. Not enough DHs submitted. at DCOfferChooseCommand.execute (/Users/cuttlefish/code/github/ot-node/modules/command/dc/dc-offer-choose-command.js:71:19) at <anonymous> 2022-10-24T00:11:27.867Z - warn - Event not sent due to releaseStage/enabledReleaseStages configuration 2022-10-24T00:11:27.867Z - info - Deleting replications directory and cache for offer with internal ID a5848007-8fcf-42b1-ae52-ec3c27063ad5

Adding Typescript to the client lib would greatly improve usability to the lib.