storjold / dataserv Goto Github PK
View Code? Open in Web Editor NEWFederated server for getting, pushing, and auditing data on untrusted nodes.
Home Page: http://storj.io
License: MIT License
Federated server for getting, pushing, and auditing data on untrusted nodes.
Home Page: http://storj.io
License: MIT License
Guessing the main problem with #49 is poor concurrency support. Switching to Postgres should solve most problems.
We currently use the ping command as a keep alive for the farmer. This way we know the farmer is available and connected. Since the farmer is already querying the node, it would be useful for the farmer to know if there are changes in to work queue. This includes contracts to store new files, or requests to transfer files back to the node.
GET /api/ping/<bitcoin address>/
RESPONSE:
Status Code: 200
Text:
Default ping delay is 15 sec. Client will send one ping every 15 sec. Server will ignore 50% of the pings.
Example:
Client send ping at sec 00
if now()- last_seen >= 15 sec then last_seen=now() -> second now() can be sec 01 and not sec 00
Next ping at sec 15
15-01 will be ignored.
I see 2 possible solution.
This will make things more unified and useful.
Some simple Exampel. max online time = 5 minutes.
What i would expect:
User1 registered 10 minutes ago and pinged 5 minutes and pinged some seconds ago. -> 100% uptime
User2 registered 10 minutes ago and pinged 5 minutes ago. -> 100% uptime.
User3 registered 10 minutes ago and pinged some seconds ago. -> 50% uptime.
User4 registered 10 minutes ago and pinged 10 minutes ago. -> 50% uptime.
User5 registered 20 minutes ago and pinged 5 minutes ago. -> 50% uptime.
User6 registered 20 minutes ago and pinged some seconds ago. -> 25% uptime.
Test results:
More unit test needed but i think there are 2 bugs.
1.) Ping
if delta.seconds <= (app.config["ONLINE_TIME"] * 60):
farmer.uptime += delta.seconds
One ping every 6 minutes will result in 0% uptime. I would expect 5 min / 6 min = 83% uptime.
User3, User5 and User6 will have this problem.
Solution:
if delta.seconds <= (app.config["ONLINE_TIME"] * 60):
farmer.uptime += delta.seconds
else:
farmer.uptime += (app.config["ONLINE_TIME"] * 60)
2.) calculate uptime
farmer_uptime = farmer.uptime + (app.config["ONLINE_TIME"] * 60)
User 5 and 6 will have a wrong uptime. After the bugfix for problem 1 farmer.uptime will be 5 Minutes cause both send 1 ping. calculate uptime will add 5 minutes more. Both will have 50% uptime. For User 6 that is wrong.
Solution:
delta = datetime.utcnow() - farmer.last_seen
if delta.seconds <= (app.config["ONLINE_TIME"] * 60):
farmer_uptime = farmer.uptime + delta.seconds
else:
farmer_uptime = farmer.uptime + (app.config["ONLINE_TIME"] * 60)
Side effect:
uptime > 100 should not be possible anymore. That was only needed cause farmer_uptime was to hight. 1 Second after the registration the uptime was 5 minutes / 1 sec = very high. With the bugfix it will be 1 sec / 1 sec.
Be carefull with solution 2. It will add a second datetime.utcnow() that can return 1 sec more than the first datetime.utcnow(). That would be a new bug for me.
I can create a pull request in 4-7h. I will add user6 as unit test. Unit test for user3 i added yesterday. The other users are not possible without faking the ping. User3 and User6 together will proof both bugs.
I am not sure and maybe i am wrong. It looks like only ping is setting uptime. Register and set height only set last seen.
The user will start the farm command. It is building and polling in one command. While it is building the server will list the client with some sec last seen but with 0% uptime.
I will write some unit test for it later.
Need a jason string for Test Group B data so we can make a pretty page to display the info, tally up all the height, and create a progress bar to our 1PB goal.
Similar to this: http://www.physikal.com/
But that page is currently just parsing the information that Shawn's page is spitting out.
The current client code here is pretty buggy, and has no unit testing or coverage.
Please add some documentation on how to run the client under the Client section of the root README.md. I spent more time that I care to admit doing --register <address>
.
shawn 9:33 PM @F483: dataserv usage is high
https://i.imgur.com/Suo5kWX.png
shawn 9:35 PM @F483: my money is on /api/online /api/online/json /api/total
they parse through every single farmer every time that is called
need to implement this on all 3 endpoints
https://pythonhosted.org/Flask-Cache/
Found locking errors in the log and a database like postgres is faster anyway.
2015-09-13 12:21:13,195 INFO app 65: CALLED /api/register/14Cro9omFYFDJGKhiGJEygZ34WQTSE9PTP/19fdPh8WN6aiWDeGjspgKZErAhqPnVVDDa
2015-09-13 12:21:15,255 ERROR dataserv.run 1423: Exception on /api/ping/17SHL6gtQXjofqgUo4WaTk4P6Q6Kwaxbxg [GET]
Traceback (most recent call last):
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/_compat.py", line 33, in reraise
raise value
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/dataserv/www/testing/dataserv/app.py", line 95, in ping
user.ping()
File "/home/dataserv/www/testing/dataserv/Farmer.py", line 145, in ping
db.session.commit()
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/scoping.py", line 150, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 790, in commit
self.transaction.commit()
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 392, in commit
self._prepare_impl()
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 372, in _prepare_impl
self.session.flush()
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2004, in flush
self._flush(objects)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2122, in _flush
transaction.rollback(_capture_exception=True)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/util/compat.py", line 182, in reraise
raise value
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2086, in _flush
flush_context.execute()
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 373, in execute
rec.execute(self)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 532, in execute
uow
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 170, in save_obj
mapper, table, update)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
execute(statement, multiparams)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
compiled_sql, distilled_params
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
exc_info
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/util/compat.py", line 188, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/util/compat.py", line 181, in reraise
raise value.with_traceback(tb)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/home/dataserv/www/testing/env/lib/python3.4/site-packages/SQLAlchemy-1.0.8-py3.4-linux-x86_64.egg/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked [SQL: 'UPDATE farmer SET last_seen=? WHERE farmer.id = ?'] [parameters: ('2015-09-13 16:21:10.243093', 165)]
Need to spec out and implement audit acceptance.
On 26th August there was a short outage as I found a fairly easy way to attack storj. A small program has been sent to @super3 showing how it was done. I prefer not to put the files here or discuss it much further so this issue is rather vague to avoid any more downtime than necessary.
This report is just to keep track of it's status.
The client needs to be able to do the following functions using the API:
We need heartbeats and IP from farmers to connect the https://live.driveshare.org/ dashboard.
Based on load the node needs to be able to rate limit the clients. Currently we are using REST like API commands for a keep-alive, and also individual audits for files. This is perhaps a poor way to do it, and we should instead have active TCP/UDP connections to each farmer. Unfortunately this would add a ton of code complexity to maintain the state of each farmer in parallel vs a simple API call.
In the meanwhile we implement simple rate limiting on the client and node side to keep things from getting overloaded. On the node we set a configuration variable, ONLINE_TIME. As long as we got a ping message from a farmer less than the number of seconds in ONLINE_TIME, we consider a farmer to be online. The client is responsible for pinging the node at some time interval less than ONLINE_TIME.
If the load state changes on the node, we need to able to communicate this with the client. So we modify the ping command as follows:
The farmer must maintain a rudimentary keep-alive with the node. This way we know if the farmer
has gone offline, and that we should not issue more challenges.
GET /api/register/<bitcoin address>/
Success Example:
GET /api/register/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE:
Status Code: 200
Text:
{
"status": "accepted",
"status-msg": None,
"timeout": 15
}
Fail Examples:
GET /api/ping/notvalidaddress/
RESPONSE:
Status Code: 400
Text:
{
"status": "rejected",
"status-msg": "Invalid BTC Address",
"timeout": 15
}
GET /api/ping/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
Status Code: 404
Text:
{
"status": "rejected",
"status-msg": "Farmer Not Found",
"timeout": 15
}
Now the node can increase or decrease expected ping times by changing ONLINE_TIME. The client should ping randomly within the range 1 second to ONLINE_TIME. This should spread out load a bit.
To keep users from hijacking other users sessions we use the blockchain to authenticate users. Each user must prove that they own their particular address. We do this by having the users sign a message (any message they choose) with their private key. This is passed to the node on every request. We can then verify that the message is valid with the Siggy library.
Users need to be warned:
@super3: flask-cache is not functioning
Example from server log
178.162.205.3 - - [31/Aug/2015 12:12:29] "GET /api/ping/1B1Re5GMSL2r3WBHmFH7z8xMY6sePrBW5M HTTP/1.1" 200 -
98.189.166.114 - - [31/Aug/2015 12:12:29] "GET /api/ping/1M8QYsZyxquxUgNjg2jvKqHHRNTEDutvvV HTTP/1.1" 404 -
98.189.166.114 - - [31/Aug/2015 12:12:31] "GET /api/ping/1Fbz4uqeDpvuLhKBgMhFJFwDfWZDCLfvvW HTTP/1.1" 404 -
81.105.145.119 - - [31/Aug/2015 12:12:31] "GET /api/ping/1Js9NtmymydFDeCPCT2evWxradh82Dfiif HTTP/1.1" 200 -
98.189.166.114 - - [31/Aug/2015 12:12:31] "GET /api/address HTTP/1.1" 200 -
Trying to connect to anything other than a local test IRC server, by modifying tests/network/test_connection.py
test_connects (tests.network.test_simultaneous_connect.TestSimultaneousConnect) ... Failed connecting to 127.0.0.1:6667 as tRdIHkGyhQnn /home/subwolf/storjnode/storjnode/network/service.py:113: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 50256)> _log.warning("Failed connecting to %s:%s as %s", host, port, nick) Couldn't connect to network! ERROR
Test server is 209.141.47.51
State "0" contracts involve files that are deterministically generated on both the node, and the farmer. This allows us to handle files, and test audit algorithms on large sets of data without having to transfer it back and forth. For example, a standard PC can generate around 100 GB of data in 30 minutes using our RandomIO library. Transferring that data directly to that farmer would take about 1 day using a standard home connection.
The standard file model should be used. The model is detailed as follows:
Farmer want to get a new contract. We want to give them a "proof of capacity" contract, also known as a state "0" contract.
GET /api/contract/new/<btc_address>
Success Example:
GET /api/contract/new/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE:
Status Code: 200
Text:
{
"btc_addr": "191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc"
"contract-type": 0,
"file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8",
"byte_size": 510,
"seed": "someseed"
}
Partial-Fail Example:
Generating state "0" contracts takes a little processing power on the node side. We have to use RandomIO to first generate the file for ourselves. If the number of clients requesting data outstrips the nodes capacity to generate this data, you will get this error.
GET /api/contract/new/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
Status Code: 102
Text: No available contracts.
Fail Example:
GET /api/contract/new/notvalidaddress/
RESPONSE:
Status Code: 400
Text: Contract Failed: Invalid BTC Address.
GET /api/contract/new/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
Status Code: 404
Text: Contract Failed: Farmer not found.
We want to know what contracts the node thinks the node the farmer should be storing.
GET /api/contract/list/<btc_address>
Success Example:
GET /api/contract/list/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE:
Status Code: 200
Text:
{
"btc_addr": "191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc"
"contracts": {
{"file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8", "state": 0},
{"file_hash": "85cd8077b4c5b306d74b6a77fe839d1bc6aecb221ab4b473134430b3a124197f", "state": 0},
}
}
Fail Example:
GET /api/contract/list/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE:
Status Code: 204
Text: No contracts.
GET /api/contract/list/notvalidaddress/
RESPONSE:
Status Code: 400
Text: Invalid BTC Address.
GET /api/contract/list/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
Status Code: 404
Text: Farmer not found.
Made a separate repo here. The client breaks my tests, and doesn't have full coverage. It should require less dependencies, and should be built on its own. Please migrate the client code to the new repo.
We want to be able to track the uptime of the farmer from registration. We do this by noting their registration time. We expect them to ping in 15 second windows, and every time they do this we increment their heartbeat by one. Therefore we calculate their uptime by the number of heartbeats they should have had since their registration.
I don't see any explicit calls to the new code. Please add a main method, and command line arguments to allow the users to run the script. User should be able to set the URL and PORT via command line. The delay will actually be set by the server later on.
Users need to be able to add files to the network. There are many farmers on the network capable of storing these files, but the user might not have the special software necessary to connect with them. Instead we upload the file to the node via a POST upload, then that node can do the proper preparations, and distribute that file to the network and farmers.
User can upload a file to the node via POST with maximum size of 32 MB.
POST /api/upload/
Success Example:
POST /api/upload/ # upload regular file
RESPONSE:
Status Code: 201
Text:
{
"file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8",
"status": "accepted"
}
Fail Example:
POST /api/upload/ # upload 33 MB file
RESPONSE:
Status Code: 413
Text:
{
"file_hash": "",
"status": "rejected"
}
If the file is found on the node, the user can download the file via GET.
GET /api/download/<file_hash>/
Success Example:
GET /api/download/c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8/
RESPONSE:
Status Code: 200
[ File Payload ]
Fail Example:
GET /api/download/invalidsha256hash/
RESPONSE:
Status Code: 400
Text: Download Failed: Invalid SHA256 Hash.
GET /api/download/f480b9e9fa9232397768df06969daa2f069d96bf3f14451a3c39b5b0e821bfe5/
RESPONSE:
Status Code: 404
Text: Download Failed: File Not Found.
The following is the model when the file is uploaded to the node.
Audit is the first call that returns HTTP code 201. Everything else returns code 200.
Clients are reporting 500 server errors and this was recorded in the log file on several occasions.
[2015-08-21 17:07:01 +0000] [5673] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 118, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 355, in import_app
import(module)
File "/root/dataserv/dataserv/app.py", line 123
total_shards += farmer.height
^
IndentationError: expected an indented block
last_seen is utcnow with milliseconds. Uptime is only integer seconds. Lets ignore the ping limit and send a ping every 2.5 seconds.
Start: last_seen = 0, uptime = 0
Ping 1: last_seen = 2.5, uptime = 2
Ping 2: last_seen = 5, uptime = 4
After the second ping we have 80% uptime. With ping limit 15.5 seconds we have 97% uptime. That is the value everyone will have at the end.
Clients are reporting 500 errors and there are no logs to figure out what’s going on.
[project:sm] Needs to be added to the root README.
Users need to be able to prove that they are storing the data they say they are storing. We do with continual audits using the PartialHash library and the Bitcoin blockchain. Using the hash of the latest Bitcoin block as the seed, we generate an audit and pass it to the node.
Farmers can only complete audits within 10 blocks of the latest block (~1 hour). Farmers can't change audits once they have submitted them.
POST /api/audit/<bitcoin address>
Payload:
Sample Payload.
Success Example:
Example.
Fail Example:
GET /api/audit/notvalidaddress
RESPONSE:
Status Code: 400
Text: Registration Failed: Invalid BTC Address.
[project:sm]
Complete this endpoint:
/api/audit/<bitcoin address>/<block_height>/<response>
The user may generate their own test data client side with RandomIO using our generation scheme. The scheme simple:
Thus using their farming address a user can deterministically generate an unlimited number of files.
Advertise Height
Allows the user to let the node know how much space they have generated via the client side generation scheme.
::
GET /api/height/<bitcoin address>/<height>/
Success Example:
::
GET /api/height/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/50/
RESPONSE:
Status Code: 200
Text: Ping Accepted.
Fail Examples:
::
GET /api/height/notvalidaddress/50/
RESPONSE:
Status Code: 400
Text: Ping Failed: Invalid BTC Address.
GET /api/height/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/50/
RESPONSE:
Status Code: 404
Text: Ping Failed: Farmer not found.
State "2" contracts involve real files that are being stored on external farmers. The standard file model should be used. The model is detailed as follows:
Farmer want to get a new contract. We pass them a state "2", so they they need to get the file before later doing audits.
GET /api/contract/new/<btc_address>
Success Example:
GET /api/contract/new/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE:
Status Code: 200
Text:
{
"btc_addr": "191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc"
"contract-type": 2,
"file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8",
"byte_size": 510
}
After success the farmer should then download the file via the API.
Partial-Fail Example:
In some cases we have no files to give to the farmers.
GET /api/contract/new/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
Status Code: 102
Text: No available contracts.
Fail Example:
GET /api/contract/new/notvalidaddress/
RESPONSE:
Status Code: 400
Text: Contract Failed: Invalid BTC Address.
GET /api/contract/new/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
Status Code: 404
Text: Contract Failed: Farmer not found.
We want to know what contracts the node thinks the node the farmer should be storing.
GET /api/contract/list/<btc_address>
Success Example:
GET /api/contract/list/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE:
Status Code: 200
Text:
{
"btc_addr": "191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc"
"contracts": {
{"file_hash": "c059c8035bbd74aa81f4c787c39390b57b974ec9af25a7248c46a3ebfe0f9dc8", "state": 2},
{"file_hash": "85cd8077b4c5b306d74b6a77fe839d1bc6aecb221ab4b473134430b3a124197f", "state": 2},
}
}
Fail Example:
GET /api/contract/list/191GVvAaTRxLmz3rW3nU5jAV1rF186VxQc/
RESPONSE:
Status Code: 204
Text: No contracts.
GET /api/contract/list/notvalidaddress/
RESPONSE:
Status Code: 400
Text: Invalid BTC Address.
GET /api/contract/list/1EawBV7n7f2wDbgxJfNzo1eHyQ9Gj77oJd/
RESPONSE:
Status Code: 404
Text: Farmer not found.
Refreshing the online list result in a random order of clients with the same height. That is confusing. A second order by payout addr would be nice.
skunk 10:49 PM how to setup dataserv server? i am getting "no such table: farmer"
shawn 10:49 PM did you setup the db?
skunk 10:51 PM i gues not
i did python app.py db init
shawn 10:51 PM got to do like a touch dataserv.db
should run nicely after that
skunk 11:31 PM nope it does not.
rm -r migrations/
python app.py db init
something missing
client set wrong payout_addr and registered.
client set a new payout_addr (without changing the wallet addr). -> register not possible because wallet addr is registered. ->build will set the height for the wrong payout_addr.
solution:
register has to check payout_addr as well. If wallet addr is registed but payout_addr changed than update the payout_addr.
Need an API call to display the addresses of all the online farmers. An online farmer is defined as someone who has completed a ping in the last 15 minutes.
To solve the issues raised by #39
[project:sm] Needs to be added to the root README.
Something to do with App.py and App, DB Objects.
Installed /home/travis/build/Storj/dataserv/.eggs/SQLAlchemy-1.0.6-py3.3-linux-x86_64.egg
running egg_info
creating dataserv.egg-info
writing dependency_links to dataserv.egg-info/dependency_links.txt
writing requirements to dataserv.egg-info/requires.txt
writing dataserv.egg-info/PKG-INFO
writing top-level names to dataserv.egg-info/top_level.txt
writing manifest file 'dataserv.egg-info/SOURCES.txt'
reading manifest file 'dataserv.egg-info/SOURCES.txt'
writing manifest file 'dataserv.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "setup.py", line 13, in <module>
test_suite="tests",
File "/opt/python/3.3.5/lib/python3.3/distutils/core.py", line 148, in setup
dist.run_commands()
File "/opt/python/3.3.5/lib/python3.3/distutils/dist.py", line 930, in run_commands
self.run_command(cmd)
File "/opt/python/3.3.5/lib/python3.3/distutils/dist.py", line 949, in run_command
cmd_obj.run()
File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/setuptools/command/test.py", line 142, in run
self.with_project_on_sys_path(self.run_tests)
File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/setuptools/command/test.py", line 122, in with_project_on_sys_path
func()
File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/setuptools/command/test.py", line 163, in run_tests
testRunner=self._resolve_as_ep(self.test_runner),
File "/opt/python/3.3.5/lib/python3.3/unittest/main.py", line 124, in __init__
self.parseArgs(argv)
File "/opt/python/3.3.5/lib/python3.3/unittest/main.py", line 168, in parseArgs
self.createTests()
File "/opt/python/3.3.5/lib/python3.3/unittest/main.py", line 175, in createTests
self.module)
File "/opt/python/3.3.5/lib/python3.3/unittest/loader.py", line 137, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/opt/python/3.3.5/lib/python3.3/unittest/loader.py", line 137, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "/opt/python/3.3.5/lib/python3.3/unittest/loader.py", line 108, in loadTestsFromName
return self.loadTestsFromModule(obj)
File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/setuptools/command/test.py", line 37, in loadTestsFromModule
tests.append(self.loadTestsFromName(submodule))
File "/opt/python/3.3.5/lib/python3.3/unittest/loader.py", line 105, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_App'
The command "coverage run --source="dataserv" setup.py test" exited with 1.
Done. Your build exited with 1.```
The channel requested that the globe display make a return. This can be done using the data published at the /api/online/json endpoint but to do geolocation the IP address of the farmer will be needed. To protect the privacy of farmers the last octet can be randomized or set to 0. If someone can add this part of the farmer IP addresses to the JSON in the next release of the server I can branch the globe display project, use the /api/online/json endpoint as a data source, and send a pull-request for the modification.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.