storj-archived / core Goto Github PK

Take the CLI program at bin/storj.js, modify it to use https://github.com/Storj/farmer (which will wrap this library), and move that functionality to https://github.com/Storj/farmer-cli

Perhaps we can still transmit file chunk in the JSON RPC message by encoding it with base91 instead of hex.

Seems promising and reduce the complexity of opening a separate connection and using a different protocol to transfer chunks.

http://base91.sourceforge.net/

Also make sure that JSDoc comments are correct and do any easy refactoring or cleanup along the way.

Create a common interface for any storage layer

Self Reporting

In order to make sure that information about nodes in the network is accurate, we ask users to voluntarily self report. For example, if a node is behind a firewall and not reachable via P2P connection, we can know that they exist and there is an issue.

Two public keys are provided. The first is used to encrypt the Node ID and IP address of the users. The second is used to encrypt the payment address. This allows a system to collect information and transparently display information without exposing the security and privacy of the users.

Keys are delivered in this format:

  { 
        "key1": hex, 
        "key2": hex
  }

The standard format for such a self reporting message is as follows:

{ 
  "store" : { 
    "free": int, 
    "used": int
  }, 
  "bandwidth": {
    "upload": float,
    "download": float
  }, 
  "node": {
    "id": key1(hex),
    "ip": key1(string),
    "port": int
  },
  "payment": key2(address), 
  "signature": sigdata 
}

Signature is of the json serialized data (minus signature) with keys ordered alphanumerically. Note that a message is considered invalid and dropped if the signature incorrect or the listed IP does not match the IP of the sender.

As normal user I would like to give my own farmer more traffic priority than the tunnel connections. That would allow me to maximize my traffic income and allow tunnel connection without the risk to loose my own better paid contracts.

As long as we don't have any traffic payment this can wait.

Currently contracts are keyed by their corresponding data hash, which makes it impossible for two renters to have a contract for the same data at the same farmer.

Create an interface for shredding a file into multiple encrypted output streams

This will involve moving some logic out of the Network class (which is much needed).

after installing node and running the install.

C:\Windows\System32>storj
path.js:8
throw new TypeError('Path must be a string. Received ' +
^

TypeError: Path must be a string. Received undefined
at assertPath (path.js:8:11)
at Object.win32.join (path.js:221:5)
at Object. (C:\Users\danm2\AppData\Roaming\npm\node_modules\storj\bin\storj.js:19:17)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Function.Module.runMain (module.js:441:10)
at startup (node.js:139:18)
at node.js:968:3

Please add a error message for wrong unlock password

root@node-storj:~# storj
 :STORJ: >> Unlock your private key to start storj >>  ********

readline.js:924
            throw err;
            ^

Error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
    at Error (native)
    at Decipher.Cipher.final (crypto.js:153:26)
    at decrypt (/root/.nvm/versions/node/v4.2.3/lib/node_modules/storj/bin/storj.js:143:18)
    at /root/.nvm/versions/node/v4.2.3/lib/node_modules/storj/bin/storj.js:173:15
    at /root/.nvm/versions/node/v4.2.3/lib/node_modules/storj/node_modules/prompt/lib/prompt.js:336:32
    at /root/.nvm/versions/node/v4.2.3/lib/node_modules/storj/node_modules/prompt/node_modules/utile/node_modules/async/lib/async.js:154:25
    at assembler (/root/.nvm/versions/node/v4.2.3/lib/node_modules/storj/node_modules/prompt/lib/prompt.js:333:9)
    at /root/.nvm/versions/node/v4.2.3/lib/node_modules/storj/node_modules/prompt/lib/prompt.js:342:32
    at /root/.nvm/versions/node/v4.2.3/lib/node_modules/storj/node_modules/prompt/lib/prompt.js:625:5
    at Interface.onLine (/root/.nvm/versions/node/v4.2.3/lib/node_modules/storj/node_modules/prompt/node_modules/read/lib/read.js:111:5)

This will help nodes determine if they are compatible with others. We can use the semver package to detect backwards incompatible versions.

Update the docs for protocol spec to reflect the changes to pubsub and CONSIGN and RETRIEVE RPC calls.

There are a number of TODOs throughout the project, let's take the time to complete them and while doing so, clean up any cruft and refactor some things. In addition, we should make the JSHint rules stricter and be sure to get the project compliant with those rules.

Document all the things, including using as a library and using the storj CLI

Driveshare-gui is not running on every system. Some users are using dataserv-client command line version. They will have to switch to node-storj command line? The self report (storj-archived/storjshare-gui#193) is required for rewards.

Self report must be implemented in node-storj or command line users are unable to get rewards.

As farmer I would like to get backup contracts. I can fill my drive in 20 days but I need 100 days to upload everything (50MBit/s down and 10MBit/s upload).
I can offer ~10MBit/s fullspeed as long as my contracts are using my drive for backups and wait more than 100 days to download them.

On the other side a farmer with only 1TB and 100MBit/s upload would like to get farmer that wait less than 1 day to download them.

An upload interval filter would allow both farmer to get max profit. Both can fill there complete drive and use there upload traffic. At the moment they will get mixed contracts. Farmer 1 will end up with an empty drive and farmer 2 will never be able to max his upload traffic.

Edit: For the renter it would be a download interval. I guess you will find a better name and a good scal.

To reduce the i/o noise for particular heavy-use farmers and renters, maybe add batch auditing requests; however this presents a challenge with request timeouts, as well as a potential spike in farmer resource utilization to address an overly long challenge list.

BinaryJS has proven to be domain specific for implementors, so let's rework it to use msgpack over a raw websocket

We need to periodically check the store_end of each contract and reap data that is no longer under contract by any renters

See storj-archived/storjshare-gui#200 (comment)

We will likely need a way for public nodes to proxy messages for firewalled or NATed nodes long term.

Something like this may work: http://reversehttp.net/relay-http-spec.html

@robertsdotpm

Non-critical but needs to happen eventually

There are many negative scenarios that are not covered by the existing test suite. Testing error cases should get our coverage much closer to 100%.

I can install node-storj fine on my own box, but the NAS is another case entirely. From my chroot I get:

root@subwolf-nas:~/node-storj# npm install .
npm WARN unmet dependency /root/node-storj/node_modules/bitcore-lib requires bs58@'=2.0.0' but will load
npm WARN unmet dependency /root/node-storj/node_modules/bs58,
npm WARN unmet dependency which is version 3.0.0
npm WARN unmet dependency /root/node-storj/node_modules/jsdoc requires async@'~1.4.0' but will load
npm WARN unmet dependency /root/node-storj/node_modules/async,
npm WARN unmet dependency which is version 1.5.2
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/module-deps requires readable-stream@'^1.1.13' but will load
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/readable-stream,
npm WARN unmet dependency which is version 2.0.5
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/crypto-browserify/node_modules/browserify-sign/node_modules/browserify-rsa requires bn.js@'^4.1.0' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/crypto-browserify/node_modules/browserify-sign/node_modules/elliptic requires bn.js@'^4.0.0' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/crypto-browserify/node_modules/diffie-hellman/node_modules/miller-rabin requires bn.js@'^4.0.0' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/crypto-browserify/node_modules/public-encrypt/node_modules/browserify-rsa requires bn.js@'^4.1.0' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/insert-module-globals/node_modules/combine-source-map/node_modules/inline-source-map requires source-map@'~0.4.0' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/labeled-stream-splicer/node_modules/stream-splicer requires readable-stream@'^1.1.13-1' but will load
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/readable-stream,
npm WARN unmet dependency which is version 2.0.5
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/labeled-stream-splicer/node_modules/stream-splicer/node_modules/readable-wrap requires readable-stream@'^1.1.13-1' but will load
npm WARN unmet dependency /root/node-storj/node_modules/browserify/node_modules/readable-stream,
npm WARN unmet dependency which is version 2.0.5

Then when trying to run the app:

root@subwolf-nas:~/node-storj# /usr/bin/nodejs bin/storj.js

module.js:340
    throw err;
          ^
Error: Cannot find module './lib/crypto/bn'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/root/node-storj/node_modules/bitcore-lib/index.js:20:21)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)

Invalid ip addresses are allowed:
0.0.0.0
127.0.0.1
192.168.1.2
255.255.255.255
Maybe a dns name if i add it to my local dns service. I didn't check it.

My public ip address is not working. I get this error message:

root@node-storj:~# storj
 :STORJ: >> Unlock your private key to start storj >>  ********

 :STORJ: {info} node created with nodeID a89b7babbd7b4071b9e6e605734223620dc301ce
 :STORJ: {info} subscribing to topic "56ce3e837f575827cb5a94e2b609756a48fa4a3882f5e762b262af31f432878d"
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: listen EADDRNOTAVAIL 91.61.66.100:4001
    at Object.exports._errnoException (util.js:874:11)
    at exports._exceptionWithHostPort (util.js:897:20)
    at Server._listen2 (net.js:1221:19)
    at listen (net.js:1270:10)
    at net.js:1379:9
    at doNTCallback3 (node.js:452:9)
    at process._tickCallback (node.js:358:17)

Create interface for taking input from multiple encrypted DataChannels and exposing a decrypted stream

Slack user blackduck and I had some bad ideas

As cheater we would like to get as many contracts as possible. Our plan to abuse tunnel traffic:

The easy way: Setup a second VM and install farmer-cli. Allow 1000 tunnel connection but don't share any drive space. The only reason for this setup is to slow down everyone else to allow my first vm to get more contracts. It is very easy to limit only the bandwidth of the second VM. My own farm VM would still run with full speed.

The radical way: Allow 1000 nodes to run fullspeed but manipulate the content. Lets drop some packages or modify them. Ofc not the first connection test message because that would invalidate the tunnel. Every other manipulation would result in a invalid signature but the tunnel would still be open right? 1000 farmer would get error messages and not any contract but they will keep the tunnel connection open to me.