I want to copy a file that is ~12M to pCloud.
To do it, I'm using the following code that is using pyfilesystem2
import fs
import urllib
from pcloud import PyCloud
import requests
import logging
import http.client as http_client
http_client.HTTPConnection.debuglevel = 1
# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
with fs.open_fs('pcloud://{username}:{password}@/'.format(
username=urllib.parse.quote_plus('***'),
password=urllib.parse.quote_plus('***'))) as pcloud_fs:
with fs.opener.open_fs('/') as linux_fs:
fs.copy.copy_file(src_fs=linux_fs,
src_path='/var/lib/fail2ban/fail2ban.sqlite3',
dst_fs=pcloud_fs,
dst_path='/fail2ban.sqlite3')
print('Completed')
DEBUG:pycloud:Doing request to https://api.pcloud.com/getdigest
DEBUG:pycloud:Params: {}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.pcloud.com:443
send: b'GET /getdigest HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:33:19 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 138
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "GET /getdigest HTTP/1.1" 200 138
DEBUG:pycloud:Doing request to https://api.pcloud.com/userinfo
DEBUG:pycloud:Params: {'getauth': 1, 'logout': 1, 'username': '***', 'digest': '***', 'passworddigest': '***'}
send: b'GET /userinfo?getauth=1&logout=1&username=***&digest=***&passworddigest=*** HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:33:19 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 744
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "GET /userinfo?getauth=1&logout=1&username=***&digest=***&passworddigest=*** HTTP/1.1" 200 744
DEBUG:pycloud:Doing request to https://api.pcloud.com/file_open
DEBUG:pycloud:Params: {'auth': '***', 'path': '/fail2ban.sqlite3', 'flags': 64}
send: b'GET /file_open?auth=***&path=%2Ffail2ban.sqlite3&flags=64 HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:33:20 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 50
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "GET /file_open?auth=***&path=%2Ffail2ban.sqlite3&flags=64 HTTP/1.1" 200 50
send: b'POST /file_write HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 2097594\r\nContent-Type: multipart/form-data; boundary=***\r\n\r\n'
send: b'--***\r\nContent-Disposition: form-data; name="fd"\r\n\r\n1\r\n--***\r\nContent-Disposition: form-data; name="data"\r\n\r\nSQLite format 3\x00\x10\x00\x01\x01\x00@ *****'
DEBUG:pycloud:Doing request to https://api.pcloud.com/file_close
DEBUG:pycloud:Params: {'auth': '***', 'fd': 1}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): api.pcloud.com:443
send: b'GET /file_close?auth=***&fd=1 HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:33:21 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 67
header: X-Error: 1007
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "GET /file_close?auth=***&fd=1 HTTP/1.1" 200 67
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1055, in _send_output
self.send(chunk)
File "/usr/local/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
File "/usr/local/lib/python3.7/ssl.py", line 1015, in sendall
v = self.send(byte_view[count:])
File "/usr/local/lib/python3.7/ssl.py", line 984, in send
return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1055, in _send_output
self.send(chunk)
File "/usr/local/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
File "/usr/local/lib/python3.7/ssl.py", line 1015, in sendall
v = self.send(byte_view[count:])
File "/usr/local/lib/python3.7/ssl.py", line 984, in send
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 76, in <module>
dst_path='/fail2ban.sqlite3')
File "/usr/local/lib/python3.7/site-packages/fs/copy.py", line 144, in copy_file
_dst_fs.upload(dst_path, read_file)
File "/usr/local/lib/python3.7/site-packages/fs/base.py", line 1313, in upload
tools.copy_file_data(file, dst_file, chunk_size=chunk_size)
File "/usr/local/lib/python3.7/site-packages/fs/tools.py", line 57, in copy_file_data
write(chunk)
File "/usr/local/lib/python3.7/site-packages/pcloud/pcloudfs.py", line 51, in write
self.pcloud.file_write(fd=self.fd, data=b)
File "/usr/local/lib/python3.7/site-packages/pcloud/validate.py", line 20, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/pcloud/api.py", line 221, in file_write
return self._upload('file_write', files, **kwargs)
File "/usr/local/lib/python3.7/site-packages/pcloud/api.py", line 112, in _upload
data=kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
This instead is what I get if I try to upload a text file that contains only "HelloWorld", where I can clearly see the urllib3.connectionpool to /file_write:
DEBUG:pycloud:Doing request to https://api.pcloud.com/getdigest
DEBUG:pycloud:Params: {}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.pcloud.com:443
send: b'GET /getdigest HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:45:17 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 138
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "GET /getdigest HTTP/1.1" 200 138
DEBUG:pycloud:Doing request to https://api.pcloud.com/userinfo
DEBUG:pycloud:Params: {'getauth': 1, 'logout': 1, 'username': '***', 'digest': '***', 'passworddigest': '***'}
send: b'GET /userinfo?getauth=1&logout=1&username=***&digest=***&passworddigest=*** HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:45:17 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 744
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "GET /userinfo?getauth=1&logout=1&username=***&digest=***&passworddigest=*** HTTP/1.1" 200 744
DEBUG:pycloud:Doing request to https://api.pcloud.com/file_open
DEBUG:pycloud:Params: {'auth': '***', 'path': '/fail2ban.sqlite3', 'flags': 64}
send: b'GET /file_open?auth=***&path=%2Ffail2ban.sqlite3&flags=64 HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:45:17 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 50
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "GET /file_open?auth=***&path=%2Ffail2ban.sqlite3&flags=64 HTTP/1.1" 200 50
send: b'POST /file_write HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 464\r\nContent-Type: multipart/form-data; boundary=***\r\n\r\n'
send: b'--***\r\nContent-Disposition: form-data; name="fd"\r\n\r\n1\r\n--***\r\nContent-Disposition: form-data; name="data"\r\n\r\nHelloWorld\n\r\n--***\r\nContent-Disposition: form-data; name="auth"\r\n\r\n***\r\n--***\r\nContent-Disposition: form-data; name="filename"; filename="filename"\r\n\r\nHelloWorld\n\r\n--***--\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:45:17 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 30
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "POST /file_write HTTP/1.1" 200 30
DEBUG:pycloud:Doing request to https://api.pcloud.com/file_close
DEBUG:pycloud:Params: {'auth': '***', 'fd': 1}
send: b'GET /file_close?auth=***&fd=1 HTTP/1.1\r\nHost: api.pcloud.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: CloudHTTPd-API v1.1
header: Date: Wed, 27 Mar 2019 20:45:17 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 16
header: ETag: "***"
header: Cache-Control: private, max-age=0
header: Vary: Accept-Encoding
header: Connection: keep-alive
header: Keep-Alive: timeout=1800
DEBUG:urllib3.connectionpool:https://api.pcloud.com:443 "GET /file_close?auth=***&fd=1 HTTP/1.1" 200 16
Completed
and as you can see the file is uploaded successfully.
Have I understood correctly the problem?
If so, would you be able to give me any hint on how I can make file_close to wait on the file_write?
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent