Guessing the main problem with #49 is poor concurrency support. Switching to Postgres should solve most problems.

We currently use the ping command as a keep alive for the farmer. This way we know the farmer is available and connected. Since the farmer is already querying the node, it would be useful for the farmer to know if there are changes in to work queue. This includes contracts to store new files, or requests to transfer files back to the node.

New API Command:

GET /api/ping/<bitcoin address>/
RESPONSE: 
   Status Code: 200
   Text: 

Default ping delay is 15 sec. Client will send one ping every 15 sec. Server will ignore 50% of the pings.

Example:
Client send ping at sec 00
if now()- last_seen >= 15 sec then last_seen=now() -> second now() can be sec 01 and not sec 00

Next ping at sec 15
15-01 will be ignored.

I see 2 possible solution.

1. new_variable=now() if new_variable-last_seen >= 15 then last_seen=new_variable -> no time difference
2. Higher default delay on the client side.

This will make things more unified and useful.

Some simple Exampel. max online time = 5 minutes.

What i would expect:
User1 registered 10 minutes ago and pinged 5 minutes and pinged some seconds ago. -> 100% uptime
User2 registered 10 minutes ago and pinged 5 minutes ago. -> 100% uptime.
User3 registered 10 minutes ago and pinged some seconds ago. -> 50% uptime.
User4 registered 10 minutes ago and pinged 10 minutes ago. -> 50% uptime.
User5 registered 20 minutes ago and pinged 5 minutes ago. -> 50% uptime.
User6 registered 20 minutes ago and pinged some seconds ago. -> 25% uptime.

Test results:
More unit test needed but i think there are 2 bugs.

1.) Ping
if delta.seconds <= (app.config["ONLINE_TIME"] * 60):
farmer.uptime += delta.seconds

One ping every 6 minutes will result in 0% uptime. I would expect 5 min / 6 min = 83% uptime.
User3, User5 and User6 will have this problem.

Solution:
if delta.seconds <= (app.config["ONLINE_TIME"] * 60):
farmer.uptime += delta.seconds
else:
farmer.uptime += (app.config["ONLINE_TIME"] * 60)

2.) calculate uptime
farmer_uptime = farmer.uptime + (app.config["ONLINE_TIME"] * 60)

User 5 and 6 will have a wrong uptime. After the bugfix for problem 1 farmer.uptime will be 5 Minutes cause both send 1 ping. calculate uptime will add 5 minutes more. Both will have 50% uptime. For User 6 that is wrong.

Solution:
delta = datetime.utcnow() - farmer.last_seen
if delta.seconds <= (app.config["ONLINE_TIME"] * 60):
farmer_uptime = farmer.uptime + delta.seconds
else:
farmer_uptime = farmer.uptime + (app.config["ONLINE_TIME"] * 60)

Side effect:
uptime > 100 should not be possible anymore. That was only needed cause farmer_uptime was to hight. 1 Second after the registration the uptime was 5 minutes / 1 sec = very high. With the bugfix it will be 1 sec / 1 sec.

Be carefull with solution 2. It will add a second datetime.utcnow() that can return 1 sec more than the first datetime.utcnow(). That would be a new bug for me.

I can create a pull request in 4-7h. I will add user6 as unit test. Unit test for user3 i added yesterday. The other users are not possible without faking the ping. User3 and User6 together will proof both bugs.

I am not sure and maybe i am wrong. It looks like only ping is setting uptime. Register and set height only set last seen.

The user will start the farm command. It is building and polling in one command. While it is building the server will list the client with some sec last seen but with 0% uptime.

I will write some unit test for it later.

Need a jason string for Test Group B data so we can make a pretty page to display the info, tally up all the height, and create a progress bar to our 1PB goal.

Similar to this: http://www.physikal.com/

But that page is currently just parsing the information that Shawn's page is spitting out.

1. bandwidth api missing in the documentation (readme)
2. bandwidth values are missing in the json online list
3. and the last point: Change the old auth address into nodeid (readme).

The current client code here is pretty buggy, and has no unit testing or coverage.

Please add some documentation on how to run the client under the Client section of the root README.md. I spent more time that I care to admit doing --register <address>.

shawn 9:33 PM @F483: dataserv usage is high
https://i.imgur.com/Suo5kWX.png

shawn 9:35 PM @F483: my money is on /api/online /api/online/json /api/total
they parse through every single farmer every time that is called
need to implement this on all 3 endpoints
https://pythonhosted.org/Flask-Cache/

Found locking errors in the log and a database like postgres is faster anyway.

2015-09-13 12:21:13,195 INFO app 65: CALLED /api/register/14Cro9omFYFDJGKhiGJEygZ34WQTSE9PTP/19fdPh8WN6aiWDeGjspgKZErAhqPnVVDDa
2015-09-13 12:21:15,255 ERROR dataserv.run 1423: Exception on /api/ping/17SHL6gtQXjofqgUo4WaTk4P6Q6Kwaxbxg [GET]
Traceback (most recent call last):
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/dataserv/www/testing/dataserv/app.py", line 95, in ping
    user.ping()
  File "/home/dataserv/www/testing/dataserv/Farmer.py", line 145, in ping
    db.session.commit()
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/scoping.py", line 150, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 790, in commit
    self.transaction.commit()
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 392, in commit
    self._prepare_impl()
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 372, in _prepare_impl
    self.session.flush()
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2004, in flush
    self._flush(objects)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2122, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/util/compat.py", line 182, in reraise
    raise value
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2086, in _flush
    flush_context.execute()
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
    execute(statement, multiparams)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    compiled_sql, distilled_params
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/util/compat.py", line 181, in reraise
    raise value.with_traceback(tb)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked [SQL: 'UPDATE farmer SET last_seen=? WHERE farmer.id = ?'] [parameters: ('2015-09-13 16:21:10.243093', 165)]

Need to spec out and implement audit acceptance.

On 26th August there was a short outage as I found a fairly easy way to attack storj. A small program has been sent to @super3 showing how it was done. I prefer not to put the files here or discuss it much further so this issue is rather vague to avoid any more downtime than necessary.

This report is just to keep track of it's status.

The client needs to be able to do the following functions using the API:

1. Attempt to register a BTC address to the node
2. Check for new data contracts on the node
3. Create new data contracts up the client or node side limit
4. Complete a keep-alive to the node.

We need heartbeats and IP from farmers to connect the https://live.driveshare.org/ dashboard.

Based on load the node needs to be able to rate limit the clients. Currently we are using REST like API commands for a keep-alive, and also individual audits for files. This is perhaps a poor way to do it, and we should instead have active TCP/UDP connections to each farmer. Unfortunately this would add a ton of code complexity to maintain the state of each farmer in parallel vs a simple API call.

In the meanwhile we implement simple rate limiting on the client and node side to keep things from getting overloaded. On the node we set a configuration variable, ONLINE_TIME. As long as we got a ping message from a farmer less than the number of seconds in ONLINE_TIME, we consider a farmer to be online. The client is responsible for pinging the node at some time interval less than ONLINE_TIME.

If the load state changes on the node, we need to able to communicate this with the client. So we modify the ping command as follows:

API Changes

Ping-Pong

The farmer must maintain a rudimentary keep-alive with the node. This way we know if the farmer
has gone offline, and that we should not issue more challenges.

GET /api/register/<bitcoin address>/

Success Example:

GET /api/register/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE:
   Status Code: 200
   Text:
       {
            "status": "accepted",
            "status-msg": None,
            "timeout": 15
       }

Fail Examples:

GET /api/ping/notvalidaddress/
RESPONSE:
    Status Code: 400
    Text:
         {
                "status": "rejected",
                "status-msg": "Invalid BTC Address",
                "timeout": 15
         }

GET /api/ping/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
    Status Code: 404
    Text:
        {
                "status": "rejected",
                "status-msg": "Farmer Not Found",
                "timeout": 15
        }

Now the node can increase or decrease expected ping times by changing ONLINE_TIME. The client should ping randomly within the range 1 second to ONLINE_TIME. This should spread out load a bit.

To keep users from hijacking other users sessions we use the blockchain to authenticate users. Each user must prove that they own their particular address. We do this by having the users sign a message (any message they choose) with their private key. This is passed to the node on every request. We can then verify that the message is valid with the Siggy library.

Users need to be warned:

1. They should not share their private key anyone.
2. They should not sign any messages on the address they use to farm.

@super3: flask-cache is not functioning

Example from server log

178.162.205.3 - - [31/Aug/2015 12:12:29] "GET /api/ping/1B1Re5GMSL2r3WBHmFH7z8xMY6sePrBW5M HTTP/1.1" 200 -
98.189.166.114 - - [31/Aug/2015 12:12:29] "GET /api/ping/1M8QYsZyxquxUgNjg2jvKqHHRNTEDutvvV HTTP/1.1" 404 -
98.189.166.114 - - [31/Aug/2015 12:12:31] "GET /api/ping/1Fbz4uqeDpvuLhKBgMhFJFwDfWZDCLfvvW HTTP/1.1" 404 -
81.105.145.119 - - [31/Aug/2015 12:12:31] "GET /api/ping/1Js9NtmymydFDeCPCT2evWxradh82Dfiif HTTP/1.1" 200 -
98.189.166.114 - - [31/Aug/2015 12:12:31] "GET /api/address HTTP/1.1" 200 -

Hello,

Can you sort:
http://104.236.104.117/api/online
By height?

Cheers,

Trying to connect to anything other than a local test IRC server, by modifying tests/network/test_connection.py

test_connects (tests.network.test_simultaneous_connect.TestSimultaneousConnect) ... Failed connecting to 127.0.0.1:6667 as tRdIHkGyhQnn /home/subwolf/storjnode/storjnode/network/service.py:113: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 50256)> _log.warning("Failed connecting to %s:%s as %s", host, port, nick) Couldn't connect to network! ERROR

Test server is 209.141.47.51

State "0" contracts involve files that are deterministically generated on both the node, and the farmer. This allows us to handle files, and test audit algorithms on large sets of data without having to transfer it back and forth. For example, a standard PC can generate around 100 GB of data in 30 minutes using our RandomIO library. Transferring that data directly to that farmer would take about 1 day using a standard home connection.

The standard file model should be used. The model is detailed as follows:

• ID - Primary key for the file in the database.
• File_Hash - SHA256 hash of the file content.
• Byte_Size - Size in bytes of the file.
• State - Storage state of the file.
  • State "0" - Deterministically generated file.
  • State "1" - Currently being stored on the node.
  • State "2" - Currently being stored on a farmer connected to the node.
• Seed - Dual use. For files with a state of "0", we use this and byte_size
  to generate the file on the fly. For files with a state of "2", we use this generate
  challenges.

API Changes

New Contract

Farmer want to get a new contract. We want to give them a "proof of capacity" contract, also known as a state "0" contract.

GET /api/contract/new/<btc_address>

Success Example:

GET /api/contract/new/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE: 
    Status Code: 200
    Text:
        {
            "btc_addr": "191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc"
            "contract-type": 0, 
            "file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8",
            "byte_size": 510,
            "seed": "someseed"
        }

Partial-Fail Example:

Generating state "0" contracts takes a little processing power on the node side. We have to use RandomIO to first generate the file for ourselves. If the number of clients requesting data outstrips the nodes capacity to generate this data, you will get this error.

GET /api/contract/new/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
    Status Code: 102
    Text: No available contracts. 

Fail Example:

GET /api/contract/new/notvalidaddress/
RESPONSE: 
    Status Code: 400 
    Text: Contract Failed: Invalid BTC Address.

GET /api/contract/new/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
    Status Code: 404
    Text: Contract Failed: Farmer not found.

List Contracts

We want to know what contracts the node thinks the node the farmer should be storing.

GET /api/contract/list/<btc_address>

Success Example:

GET /api/contract/list/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE: 
    Status Code: 200
    Text:
        {
            "btc_addr": "191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc"
            "contracts": {
                {"file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8", "state": 0},
                {"file_hash": "85cd8077b4c5b306d74b6a77fe839d1bc6aecb221ab4b473134430b3a124197f", "state": 0},
            }
        }

Fail Example:

GET /api/contract/list/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE: 
    Status Code: 204
    Text: No contracts.

GET /api/contract/list/notvalidaddress/
RESPONSE: 
    Status Code: 400 
    Text: Invalid BTC Address.

GET /api/contract/list/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
    Status Code: 404
    Text: Farmer not found.           

Made a separate repo here. The client breaks my tests, and doesn't have full coverage. It should require less dependencies, and should be built on its own. Please migrate the client code to the new repo.

We want to be able to track the uptime of the farmer from registration. We do this by noting their registration time. We expect them to ping in 15 second windows, and every time they do this we increment their heartbeat by one. Therefore we calculate their uptime by the number of heartbeats they should have had since their registration.

I don't see any explicit calls to the new code. Please add a main method, and command line arguments to allow the users to run the script. User should be able to set the URL and PORT via command line. The delay will actually be set by the server later on.

Users need to be able to add files to the network. There are many farmers on the network capable of storing these files, but the user might not have the special software necessary to connect with them. Instead we upload the file to the node via a POST upload, then that node can do the proper preparations, and distribute that file to the network and farmers.

API Changes

Upload

User can upload a file to the node via POST with maximum size of 32 MB.

POST /api/upload/

Success Example:

POST /api/upload/  # upload regular file
RESPONSE:
    Status Code: 201
    Text:
        {
            "file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8",
            "status": "accepted"
        }

Fail Example:

POST /api/upload/  # upload 33 MB file
RESPONSE:
    Status Code: 413
    Text:
        {
            "file_hash": "",
            "status": "rejected"
        }

Download

If the file is found on the node, the user can download the file via GET.

GET /api/download/<file_hash>/

Success Example:

GET /api/download/c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8/    
RESPONSE:
    Status Code: 200
    [ File Payload ]

Fail Example:

GET /api/download/invalidsha256hash/
RESPONSE:
    Status Code: 400
    Text: Download Failed: Invalid SHA256 Hash.

GET /api/download/f480b9e9fa9232397768df06969daa2f069d96bf3f14451a3c39b5b0e821bfe5/
RESPONSE:
    Status Code: 404
    Text: Download Failed: File Not Found.

The following is the model when the file is uploaded to the node.

• ID - Primary key for the file in the database.
• File_Hash - SHA256 hash of the file content.
• Byte_Size - Size in bytes of the file.
• State - Storage state of the file.
  • State "0" - Deterministically generated file.
  • State "1" - Currently being stored on the node.
  • State "2" - Currently being stored on a farmer connected to the node.
• Seed - Dual use. For files with a state of "0", we use this and byte_size
  to generate the file on the fly. For files with a state of "2", we use this generate
  challenges.

Audit is the first call that returns HTTP code 201. Everything else returns code 200.

Clients are reporting 500 server errors and this was recorded in the log file on several occasions.

[2015-08-21 17:07:01 +0000] [5673] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 118, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 355, in import_app
import(module)
File "/root/dataserv/dataserv/app.py", line 123
total_shards += farmer.height
^
IndentationError: expected an indented block

last_seen is utcnow with milliseconds. Uptime is only integer seconds. Lets ignore the ping limit and send a ping every 2.5 seconds.

Start: last_seen = 0, uptime = 0
Ping 1: last_seen = 2.5, uptime = 2
Ping 2: last_seen = 5, uptime = 4

After the second ping we have 80% uptime. With ping limit 15.5 seconds we have 97% uptime. That is the value everyone will have at the end.

Clients are reporting 500 errors and there are no logs to figure out what’s going on.

[project:sm] Needs to be added to the root README.

Block Audit

Users need to be able to prove that they are storing the data they say they are storing. We do with continual audits using the PartialHash library and the Bitcoin blockchain. Using the hash of the latest Bitcoin block as the seed, we generate an audit and pass it to the node.

Farmers can only complete audits within 10 blocks of the latest block (~1 hour). Farmers can't change audits once they have submitted them.

POST /api/audit/<bitcoin address>

Payload:

Sample Payload.

Success Example:

Example.

Fail Example:

GET /api/audit/notvalidaddress
RESPONSE:
    Status Code: 400
    Text: Registration Failed: Invalid BTC Address.

[project:sm]
Complete this endpoint:
/api/audit/<bitcoin address>/<block_height>/<response>

The user may generate their own test data client side with RandomIO using our generation scheme. The scheme simple:

• User takes the SHA-256 hash of their BTC address.
• That value is that seed, and the SHA-256 hash of that seed is the next value, etc.
• Seeds are passed into RandomIO with byte_size of 128 MB
• File is saved with its hash as its file name

Thus using their farming address a user can deterministically generate an unlimited number of files.

Advertise Height

Allows the user to let the node know how much space they have generated via the client side generation scheme.

::

GET /api/height/<bitcoin address>/<height>/

Success Example:

::

GET /api/height/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/50/
RESPONSE:
   Status Code: 200
   Text: Ping Accepted.

Fail Examples:

::

GET /api/height/notvalidaddress/50/
RESPONSE:
    Status Code: 400
    Text: Ping Failed: Invalid BTC Address.

GET /api/height/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/50/
RESPONSE:
    Status Code: 404
    Text: Ping Failed: Farmer not found.

State "2" contracts involve real files that are being stored on external farmers. The standard file model should be used. The model is detailed as follows:

• ID - Primary key for the file in the database.
• File_Hash - SHA256 hash of the file content.
• Byte_Size - Size in bytes of the file.
• State - Storage state of the file.
  • State "0" - Deterministically generated file.
  • State "1" - Currently being stored on the node.
  • State "2" - Currently being stored on a farmer connected to the node.
• Seed - Dual use. For files with a state of "0", we use this and byte_size
  to generate the file on the fly. For files with a state of "2", we use this generate
  challenges.

API Changes

New Contract

Farmer want to get a new contract. We pass them a state "2", so they they need to get the file before later doing audits.

GET /api/contract/new/<btc_address>

Success Example:

GET /api/contract/new/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE: 
    Status Code: 200
    Text:
        {
            "btc_addr": "191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc"
            "contract-type": 2, 
            "file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8",
            "byte_size": 510
        }

After success the farmer should then download the file via the API.

Partial-Fail Example:

In some cases we have no files to give to the farmers.

GET /api/contract/new/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
    Status Code: 102
    Text: No available contracts. 

Fail Example:

GET /api/contract/new/notvalidaddress/
RESPONSE: 
    Status Code: 400 
    Text: Contract Failed: Invalid BTC Address.

GET /api/contract/new/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
    Status Code: 404
    Text: Contract Failed: Farmer not found.

List Contracts

We want to know what contracts the node thinks the node the farmer should be storing.

GET /api/contract/list/<btc_address>

Success Example:

GET /api/contract/list/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE: 
    Status Code: 200
    Text:
        {
            "btc_addr": "191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc"
            "contracts": {
                {"file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8", "state": 2},
                {"file_hash": "85cd8077b4c5b306d74b6a77fe839d1bc6aecb221ab4b473134430b3a124197f", "state": 2},
            }
        }

Fail Example:

GET /api/contract/list/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE: 
    Status Code: 204
    Text: No contracts.

GET /api/contract/list/notvalidaddress/
RESPONSE: 
    Status Code: 400 
    Text: Invalid BTC Address.

GET /api/contract/list/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
    Status Code: 404
    Text: Farmer not found.           

Refreshing the online list result in a random order of clients with the same height. That is confusing. A second order by payout addr would be nice.

skunk 10:49 PM how to setup dataserv server? i am getting "no such table: farmer"

shawn 10:49 PM did you setup the db?

skunk 10:51 PM i gues not
i did python app.py db init

shawn 10:51 PM got to do like a touch dataserv.db
should run nicely after that

skunk 11:31 PM nope it does not.
rm -r migrations/
python app.py db init
something missing

client set wrong payout_addr and registered.
client set a new payout_addr (without changing the wallet addr). -> register not possible because wallet addr is registered. ->build will set the height for the wrong payout_addr.

solution:
register has to check payout_addr as well. If wallet addr is registed but payout_addr changed than update the payout_addr.

• btc_addr need to be changed to a node_id compatibale with MetaDisk
• add bandwidth field
• add ip field(we will geolocate this later)

Need an API call to display the addresses of all the online farmers. An online farmer is defined as someone who has completed a ping in the last 15 minutes.

To solve the issues raised by #39

[project:sm] Needs to be added to the root README.

Something to do with App.py and App, DB Objects.

Installed /home/travis/build/Storj/dataserv/.eggs/SQLAlchemy-1.0.6-py3.3-linux-x86_64.egg
running egg_info
creating dataserv.egg-info
writing dependency_links to dataserv.egg-info/dependency_links.txt
writing requirements to dataserv.egg-info/requires.txt
writing dataserv.egg-info/PKG-INFO
writing top-level names to dataserv.egg-info/top_level.txt
writing manifest file 'dataserv.egg-info/SOURCES.txt'
reading manifest file 'dataserv.egg-info/SOURCES.txt'
writing manifest file 'dataserv.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 13, in <module>
    test_suite="tests",
  File "/opt/python/3.3.5/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/python/3.3.5/lib/python3.3/distutils/dist.py", line 930, in run_commands
    self.run_command(cmd)
  File "/opt/python/3.3.5/lib/python3.3/distutils/dist.py", line 949, in run_command
    cmd_obj.run()
  File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/setuptools/command/test.py", line 142, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/setuptools/command/test.py", line 122, in with_project_on_sys_path
    func()
  File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/setuptools/command/test.py", line 163, in run_tests
    testRunner=self._resolve_as_ep(self.test_runner),
  File "/opt/python/3.3.5/lib/python3.3/unittest/main.py", line 124, in __init__
    self.parseArgs(argv)
  File "/opt/python/3.3.5/lib/python3.3/unittest/main.py", line 168, in parseArgs
    self.createTests()
  File "/opt/python/3.3.5/lib/python3.3/unittest/main.py", line 175, in createTests
    self.module)
  File "/opt/python/3.3.5/lib/python3.3/unittest/loader.py", line 137, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/opt/python/3.3.5/lib/python3.3/unittest/loader.py", line 137, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/opt/python/3.3.5/lib/python3.3/unittest/loader.py", line 108, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/setuptools/command/test.py", line 37, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/opt/python/3.3.5/lib/python3.3/unittest/loader.py", line 105, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_App'
The command "coverage run --source="dataserv" setup.py test" exited with 1.
Done. Your build exited with 1.```

The channel requested that the globe display make a return. This can be done using the data published at the /api/online/json endpoint but to do geolocation the IP address of the farmer will be needed. To protect the privacy of farmers the last octet can be randomized or set to 0. If someone can add this part of the farmer IP addresses to the JSON in the next release of the server I can branch the globe display project, use the /api/online/json endpoint as a data source, and send a pull-request for the modification.