storjold / pyp2p Goto Github PK

View Code? Open in Web Editor NEW

167.0 167.0 45.0 511 KB

Home Page: http://storj.io

License: MIT License

Python 94.41% Shell 0.34% PHP 4.55% Makefile 0.70%

pyp2p's People

Contributors

Stargazers

Watchers

Forkers

pombredanne f483 littleskunk debunge runur ohadcn xer0xe9 louis0723 thanhan7914 augustand unoffices tsingjyujing owenbupt zzlettle vischub bevanh aurora81 jianzfb mofei168888 csunny j-usti-n codesfork yuyongpeng shreyanshs7 faical-yannick-congo hasanatkazmi aeroju chenxyzj kreastr emiamar pbadeer andrew4699 hummans yasmasatakemura-bf sadernalwis it9good robertsdotpm dreasion odorouscream tacitogrc sabixr longpp17

pyp2p's Issues

Fix logging!

pyp2p is spamming the log with debug statements, which make it difficult to debug using a command line debugger.

https://travis-ci.org/Storj/storjnode/builds/103261505#L1334

Loger levels should only be set according to --verbose, --debug, --quiet arguments. Having a configurable logger makes no sense if it cannot be controlled.

Register may fail in dht msg

Assert register is true otherwise the code will be in an unknown state

Assertion error when running multiple farmers.

(env)bootstrap@bootstrap-eu01:~/storjnode$ python scripts/start_farmers.py
Starting farmer
2016-03-04 14:05:25,618 ERROR storjnode.api 83: AssertionError()
Traceback (most recent call last):
  File "/home/bootstrap/storjnode/env/local/lib/python2.7/site-packages/storjnode-0.1.2-py2.7.egg/storjnode/api.py", line 80, in _init_node
    config=self._cfg
  File "/home/bootstrap/storjnode/env/local/lib/python2.7/site-packages/storjnode-0.1.2-py2.7.egg/storjnode/network/node.py", line 122, in __init__
    network_id=self.network_id
  File "/home/bootstrap/storjnode/env/local/lib/python2.7/site-packages/pyp2p/dht_msg.py", line 83, in __init__
    assert(self.is_mutex_ready.is_set())
AssertionError
Traceback (most recent call last):
  File "scripts/start_farmers.py", line 70, in <module>
    main()
  File "scripts/start_farmers.py", line 53, in main
    node = storjnode.api.StorjNode(wallet=wallet, config=config)
  File "/home/bootstrap/storjnode/env/local/lib/python2.7/site-packages/storjnode-0.1.2-py2.7.egg/storjnode/api.py", line 43, in __init__
    self._init_node()
  File "/home/bootstrap/storjnode/env/local/lib/python2.7/site-packages/storjnode-0.1.2-py2.7.egg/storjnode/api.py", line 80, in _init_node
    config=self._cfg
  File "/home/bootstrap/storjnode/env/local/lib/python2.7/site-packages/storjnode-0.1.2-py2.7.egg/storjnode/network/node.py", line 122, in __init__
    network_id=self.network_id
  File "/home/bootstrap/storjnode/env/local/lib/python2.7/site-packages/pyp2p/dht_msg.py", line 83, in __init__
    assert(self.is_mutex_ready.is_set())
AssertionError

Args overwriting the help section of other modules

Tests Delete tests/upnpc-static.exe on Windows

Tests will fail on all other tries other than the first try.

Port-Forwarding a server started using native python socket library

Hi, guys! I have been trying to find such a tool since a long time. But I am a bit confused from the docs, please have a look below.

I start a server using the native library:

def register_audio_port(self):
        '''register audio socket and start thread for receiving connections'''
        self.sock_audio = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.sock_audio.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.sock_audio.bind(('0.0.0.0',0)) # 0.0.0.0 will allow all connections
        audio_port=self.sock_audio.getsockname()[1]
        self.sock_audio.listen(75)

How can I port-forward using your library on a already started server?

Test blocking timeout error

Test blocking timeout in sock tests fails with:
print("Manual timeout failed.")
assert 0

Indicating the timeout wasn't respected.
To do: investigate this.
Priority: low (since not infinite blocking)

Update UNL for con reuse for when reverse connect code is used. Take different actions depending on whether nonces are set. This will optimize the reverse connect code so multiple connections aren't triggered by both sides with force_master trying to connect.

Strange infinite blocking error under network loads

Under periods of large network congestion and latency the person uploading data will eventually fill up their send buffer, usually only transferring a small amount of data until it errors out with "resource unavailable" - (doing this forever) which indicates it would block. The strange thing is: the receiving side -also- has the same error and only manages to receive a small amount compared to the person uploading (they eventually go out of synch: upload, say 10,000 bytes, download 2000.) So then, what is happening with the data? I almost think it's a network problem but TCP should handle something like this.

Make rendezvous servers more resilient

The current design works well for multiple servers used to bootstrap a standard flooding-based p2p network (just add servers proportional to network size, have the nodes split across servers and randomly select servers) but for hole punching all nodes need to be on the same server. Think of a way that allows this to scale.

(Won't be used for current client as it will likely be replaced with something else but if its an important issue to note.)

Use standard python logger for logging.

This is completely spamming all messages (not only the ones concerning itself).

You should use a normal python logger instead of rolling your own.

This will allow others using your lib to have more control over logging.

Improve bootstrapping design

Currently there is only 1 rendezvous server. Increase it to 2 for fault tolerance and research alternative bootstrapping means if everything fails.

Fix tests and update pyp2p.dht_msg.DHT api.

I changed the storjnode api so that direct messages are removed and the message handlers no longer get the source node id (always unknown with relay messages). pyp2p.dht_msg.DHT needs to be updated to reflect this change.

See StorjOld/storjnode@a7bee2c#diff-f02d8d10fcbce54955b651de4d023a1cR30

The current master branch however does not build (why is the develop branch so far behind?) any maybe its better if @robertsdotpm takes a look at it as he knows this code well.

======================================================================
ERROR: test_0000001_sock (tests.test_sock.test_sock)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_sock.py", line 249, in test_0000001_sock
    blocking=1)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 77, in __init__
    self.connect(addr, port)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 187, in connect
    raise socket.error("Socket connect failed.")
error: Socket connect failed.

======================================================================
ERROR: test_blocking_timeout (tests.test_sock.test_sock)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_sock.py", line 193, in test_blocking_timeout
    s = Sock(rendezvous_servers[0]["addr"], rendezvous_servers[0]["port"], blocking=1)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 77, in __init__
    self.connect(addr, port)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 187, in connect
    raise socket.error("Socket connect failed.")
error: Socket connect failed.

======================================================================
ERROR: test_encoding (tests.test_sock.test_sock)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_sock.py", line 208, in test_encoding
    s = Sock(rendezvous_servers[0]["addr"], rendezvous_servers[0]["port"], blocking=1)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 77, in __init__
    self.connect(addr, port)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 187, in connect
    raise socket.error("Socket connect failed.")
error: Socket connect failed.

======================================================================
ERROR: test_non_blocking_timeout (tests.test_sock.test_sock)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_sock.py", line 202, in test_non_blocking_timeout
    s = Sock(rendezvous_servers[0]["addr"], rendezvous_servers[0]["port"], blocking=0)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 77, in __init__
    self.connect(addr, port)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 187, in connect
    raise socket.error("Socket connect failed.")
error: Socket connect failed.

======================================================================
ERROR: test_recv_recvline_switch (tests.test_sock.test_sock)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_sock.py", line 221, in test_recv_recvline_switch
    s = Sock(rendezvous_servers[0]["addr"], rendezvous_servers[0]["port"], blocking=1)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 77, in __init__
    self.connect(addr, port)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 187, in connect
    raise socket.error("Socket connect failed.")
error: Socket connect failed.

======================================================================
ERROR: test_00001 (tests.test_rendezvous_server.test_rendezvous_client)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_rendezvous_server.py", line 13, in test_00001
    rendezvous_servers[0]["port"], blocking=1)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 77, in __init__
    self.connect(addr, port)
  File "/home/f483/dev/storj/pyp2p/pyp2p/sock.py", line 187, in connect
    raise socket.error("Socket connect failed.")
error: Socket connect failed.

======================================================================
FAIL: test_00000002 (tests.test_net.test_net)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_net.py", line 566, in test_00000002
    assert(0)
AssertionError

======================================================================
FAIL: test_nat_tcp_hole_punch (tests.test_net.test_net)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_net.py", line 56, in test_nat_tcp_hole_punch
    assert(0)
AssertionError

======================================================================
FAIL: test_net_config (tests.test_net.test_net)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/f483/dev/storj/pyp2p/tests/test_net.py", line 348, in test_net_config
    assert(conf_out == found)
AssertionError

----------------------------------------------------------------------

Cache network type info

error when pip install -r requirements.txt

Problem:

Although i installed pip 7.1.2 in my virtualenv, Error is raised when i use
pip install -r requirements.txt.

System:

OSX10.10.5

Python Version:

2.7.10(using virtualenv) & pip 7.1.2

Error message:

Exception: Traceback (most recent call last): File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/basecommand.py", line 211, in main status = self.run(options, args) File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/commands/install.py", line 282, in run wheel_cache File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/basecommand.py", line 291, in populate_requirement_set wheel_cache=wheel_cache): File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/req/req_file.py", line 89, in parse_requirements for req in req_iter: File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/req/req_file.py", line 137, in process_line isolated=isolated, options=req_options, wheel_cache=wheel_cache File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/req/req_install.py", line 213, in from_line wheel_cache=wheel_cache, constraint=constraint) File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/req/req_install.py", line 67, in __init__ req = pkg_resources.Requirement.parse(req) File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2980, in parse reqs = list(parse_requirements(s)) File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2924, in parse_requirements "version spec") File "/Users/jason/opensource/pyp2p/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2889, in scan_list raise RequirementParseError(msg, line, "at", line[p:]) RequirementParseError: Expected version spec in pip 7.1.2 at 7.1.2

Solution:

I remove pip 7.1.2 and then everything go on smoothly.

Please Resurrect This Project!

I've been looking for a great P2P library for Python. None exist. Whenever I try to use this library though, it has a URLOPEN error.

Sim open != thread safe

Multiple concurrent TCP hole punches won't work. The UNL code needs to queue them

Exception: Unable to connect to rendezvous server.

I try to use this plugin in python but it just doesn't work. I get this error. Even the sample code doesn't work.

Is this project dead?

I see that this project is stored under: StorjOld/pyp2p

Does this mean that this project is no longer being worked on? (it also looks like no code updates have been provided for the past 2 years)

Exception in dht msg list - No JSON object could be decoded

Tested with linux storjnode develop branch and pyp2p 0.8.0
Same issue with the windows binary https://ci.appveyor.com/project/Storj/storjnode/build/0.1.1.13

Pyp2p don't stop spamming the logfile. The following block comes in a loop:

unknown exception
No JSON object could be decoded
EXCEPTION IN DHT MSG LIST
Exception in dht msg list
No JSON object could be decoded
EXCEPTION IN DHT MSG LIST
Exception in dht msg list
No JSON object could be decoded
EXCEPTION IN DHT MSG LIST
Exception in dht msg list
No JSON object could be decoded
EXCEPTION IN DHT MSG LIST
Exception in dht msg list
No JSON object could be decoded
EXCEPTION IN DHT MSG LIST
Exception in dht msg list
No JSON object could be decoded

More PEP8

TCP relaying

In cases where everything fails to get a connection a P2P relay design might work with fallback on a rendezvous server. Consider whether this is a good idea to use for this particular design (it might not be.)

dataserv-client 2.1.12 crashes

I decided to upgrade from dataserv-client 2.1.4 to the latest release (2.1.12). Though everything with 2.1.4 worked fine, 2.1.12 crashes immediately, even when checking the version.

I am running OSX 10.11.2, and have tried to uninstall/install/upgrade, all with the same result.

% dataserv-client version
Traceback (most recent call last):
  File "/usr/local/bin/dataserv-client", line 6, in <module>
    from dataserv_client import cli
  File "/usr/local/lib/python2.7/site-packages/dataserv_client/cli.py", line 4, in <module>
    from dataserv_client import api
  File "/usr/local/lib/python2.7/site-packages/dataserv_client/api.py", line 7, in <module>
    import storjnode
  File "/usr/local/lib/python2.7/site-packages/storjnode/__init__.py", line 3, in <module>
    from . import network  # NOQA
  File "/usr/local/lib/python2.7/site-packages/storjnode/network/__init__.py", line 1, in <module>
    from . import node  # NOQA
  File "/usr/local/lib/python2.7/site-packages/storjnode/network/node.py", line 13, in <module>
    from pyp2p.unl import UNL
  File "/usr/local/lib/python2.7/site-packages/pyp2p/unl.py", line 6, in <module>
    from .lib import *  # FIXME * is evil!
  File "/usr/local/lib/python2.7/site-packages/pyp2p/lib.py", line 37, in <module>
    ip = IPDB()
  File "/usr/local/lib/python2.7/site-packages/pyroute2/__init__.py", line 89, in __new__
    cls.register(cls.registry['class'])
  File "/usr/local/lib/python2.7/site-packages/pyroute2/__init__.py", line 69, in __getitem__
    [name], 0)
  File "/usr/local/lib/python2.7/site-packages/pyroute2/ipdb/__init__.py", line 396, in <module>
    from pyroute2.iproute import IPRoute
  File "/usr/local/lib/python2.7/site-packages/pyroute2/iproute.py", line 109, in <module>
    from pyroute2.netlink.rtnl.tcmsg import get_htb_parameters
  File "/usr/local/lib/python2.7/site-packages/pyroute2/netlink/rtnl/tcmsg.py", line 29, in <module>
    _psched = open('/proc/net/psched', 'r')
IOError: [Errno 2] No such file or directory: '/proc/net/psched'

Master branch fails to pass tests.

The master branch fails to pass test on my machine (standard ubuntu/mint linux), see test log for more info.

Hardcoded rendezvous and forwarding servers?

Is there a doc or explanation of why these are hard coded? How do I change them without editing net.py?

Also, what is net.php?

Thanks!

Improve logging

Tested with storjnode develop branch and pyp2p 0.8.0
Same problem with the windows binary (https://ci.appveyor.com/project/Storj/storjnode/build/0.1.1.13)

Expected log format:

2016-03-03 19:17:45,708 DEBUG storjnode.network.monitor 529: started monitor thread

pyp2p log output: It is not easy to find bugs without timestamp and line number.

EXCEPTION IN DHT MSG LIST
Exception in dht msg list
HTTPConnectionPool(host='162.243.213.95', port=80): Read timed out. (read timeout=5)
unknown exception
No JSON object could be decoded

Add travis and coverage badges.

Errored UNL.connect threads could stall entire build

If an error occurs on the UNL thread it may not close properly and net.stop won't kill the zombie threads, delaying the process from ending -- this could also break tests

the success handlers could also cause problems if there's exceptions in them when getting the mutex

Nodes should reconnect if rendezvous server goes down

Investigate why test nodes don't do this

Futures not in pypi

from future.moves.urllib.request import urlopen

ImportError: No module named future.moves.urllib.request

Priority: low

OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

When I run rendzevous_server.py code , I get the foolowing error . Can someone help me to fix this ?

OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted
Thanks in advance

Fix Upper Bound for Ports

Currently its set to 49151, but it should be 65535

Places where it needs to be changed:

pyp2p/lib.py:50:    if port is None or port < 1024 or port > 49151:
pyp2p/lib.py:51:        port = random.randint(1024, 49151)

Connection refused

Tested with storjnode develop branch and pyp2p 0.8.0
Windows don't has this problem.

I am running storjnode on my linux NAS in a LXC "VM". The reason for the error could be the firewall of the NAS. I don't know if the NAS comes with a simple NAT firewall.

less error.log

  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
 <class 'socket.error'> sock.py 195 [Errno 111] Connection refused
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/pyp2p/sock.py", line 195, in connect
    self.s.connect((addr, int(port)))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
 <class 'socket.error'> sock.py 195 [Errno 111] Connection refused
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/pyp2p/sock.py", line 195, in connect
    self.s.connect((addr, int(port)))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
 <class 'socket.error'> sock.py 195 [Errno 111] Connection refused
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/pyp2p/sock.py", line 195, in connect
    self.s.connect((addr, int(port)))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
 <class 'socket.error'> sock.py 195 [Errno 111] Connection refused
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/pyp2p/net.py", line 654, in determine_node
    lan_ip)
  File "build/bdist.linux-x86_64/egg/pyp2p/upnp.py", line 203, in forward_port
    raise Exception("Unable to find UPnP compatible gateway.")
Exception: Unable to find UPnP compatible gateway.
 <type 'exceptions.Exception'> net.py 654 Unable to find UPnP compatible gateway.
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/pyp2p/net.py", line 674, in determine_node
    lan_ip)
  File "build/bdist.linux-x86_64/egg/pyp2p/nat_pmp.py", line 482, in forward_port
    return map_port(proto, src_port, dest_port)
  File "build/bdist.linux-x86_64/egg/pyp2p/nat_pmp.py", line 413, in map_port
    retry=retry)
  File "build/bdist.linux-x86_64/egg/pyp2p/nat_pmp.py", line 454, in send_request_with_retry
    error_str(NATPMP_GATEWAY_NO_SUPPORT))
NATPMPUnsupportedError: (-11, 'The gateway does not support NAT-PMP')
 <class 'pyp2p.nat_pmp.NATPMPUnsupportedError'> net.py 674 (-11, 'The gateway does not support NAT-PMP')

Psutil not in pypi

import psutil

ImportError: No module named psutil

Branch isn't up to date or requirements are being ignored.