lukasa / requests-ftp

An FTP transport adapter for use with the Python Requests library.

Requests-FTP

Requests-FTP is an implementation of a very stupid FTP transport adapter for use with the awesome Requests Python library.

This library is not intended to be an example of Transport Adapters best practices. This library was cowboyed together in about 4 hours of total work, has no tests, and relies on a few ugly hacks. Instead, it is intended as both a starting point for future development and a useful example for how to implement transport adapters.

Here's how you use it:

>>> import requests_ftp
>>> s = requests_ftp.FTPSession()
>>> resp = s.list('ftp://127.0.0.1/', auth=('Lukasa', 'notmypass'))
>>> resp.status_code
'226'
>>> print resp.content
...snip...
>>> resp = s.stor('ftp://127.0.0.1/test.txt', auth=('Lukasa', 'notmypass'),
                   files={'file': open('report.txt', 'rb')})

Features

Almost none!

• Adds the FTP LIST, STOR, RETR and NLST verbs via a new FTP transport adapter.
• Provides a function that monkeypatches the Requests Session object, exposing helper methods much like the current Session.get() and Session.post() methods.
• Piggybacks on standard Requests idioms: uses normal Requests models and access methods, including the tuple form of authentication.

Does not provide:

• Connection pooling! One new connection and multiple commands for each request, including authentication. Super inefficient.
• SFTP. Security is for the weak.
• Less common commands.

Monkey Patching

Sometimes you may want to call a library that uses requests with an ftp URL. First, check whether the library takes a session parameter. If it does, you can use either the FTPSession or FTPAdapter class directly, which is the preferred approach:

>>> import requests_ftp
>>> import some_library
>>> s = requests_ftp.FTPSession()
>>> resp = some_library.get('ftp://127.0.0.1/', auth=('Lukasa', 'notmypass'), session=s)

If they do not, either modify the library to add a session parameter, or as an absolute last resort, use the monkeypatch_session function:

>>> import requests_ftp
>>> requests_ftp.monkeypatch_session()
>>> import some_library
>>> resp = some_library.get('ftp://127.0.0.1/', auth=('Lukasa', 'notmypass'))

If you expect your code to be used as a library, take particular care to avoid the monkeypatch_session option.

Important Notes

Many corners have been cut in my rush to get this code finished. The most obvious problem is that this code does not have any tests. This is my highest priority for fixing.

More notably, we have the following important caveats:

• The design of the Requests Transport Adapater means that the STOR method has to un-encode a multipart form-data encoded body to get the file. This is painful, and I haven't tested this thoroughly, so it might not work.
• Massive assumptions have been made in the use of the STOR method. This code assumes that there will only be one file included in the files argument. It also requires that you provide the filename to save as as part of the URL. This is single-handedly the most brittle part of this adapter.
• This code is not optimised for performance AT ALL. There is some low-hanging fruit here: we should be able to connection pool relatively easily, and we can probably avoid making some of the requests we do.

Contributing

Please do! I would love for this to be developed further by anyone who is interested. Wherever possible, please provide unit tests for your work (yes, this is very much a 'do as I say, not as I do' kind of moment). Don't forget to add your name to AUTHORS.

License

To maximise compatibility with Requests, this code is licensed under the Apache license. See LICENSE for more details.

requests-ftp's Issues

requests-ftp maintenance

hey @Lukasa

You say in #10 (comment)

@adrienbrunet This library is definitely far from perfection: it was mostly intended as a joke when I originally wrote it! I'm definitely open to finding someone else to take over maintenance of it and to turn it into something more useful, so if you feel like proposing some pull requests and getting this library up and running into something better, I'll happily consider taking on a maintainer.

What does that mean in immediate practical terms? Do you think it should be rewritten to be more .. in line with whatever best practices there are for requests transport adapters? And some tests.

I think it's a very useful obvious hole for those just looking to interact with FTP in ~2 lines. I'm not offering to be a maintainer since for me that just means trying to do what's needed and make lots of PRs, and if I can make the time for that then presto.

ValueError: invalid literal for int() with base 10: '226-Options:'

import requests
import requests_ftp
requests_ftp.monkeypatch_session()
s = requests.Session()
resp = s.list('ftp://host/', auth=('user', 'password'))
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.6/site-packages/requests_ftp/ftp.py", line 25, in list
return self.request('LIST', url, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.6/site-packages/requests_ftp/ftp.py", line 182, in send
resp = self.func_table[request.method](path, request)
File "/usr/lib/python3.6/site-packages/requests_ftp/ftp.py", line 245, in list
response = build_text_response(request, data, code)
File "/usr/lib/python3.6/site-packages/requests_ftp/ftp.py", line 97, in build_text_response
return build_response(request, data, code, 'ascii')
File "/usr/lib/python3.6/site-packages/requests_ftp/ftp.py", line 116, in build_response
response.status_code = int(code.split()[0])
ValueError: invalid literal for int() with base 10: '226-Options:'

cgi module is deprecated and removed in Python 3.13

Affected code:

requests-ftp/requests_ftp/ftp.py

Line 68 in d959118

_, pdict = cgi.parse_header(request.headers['Content-Type'])

See what's new in 3.13 for possible alternatives.

fails if the host does not endswith a /

This works:

>>> s = requests.Session()
>>> resp = s.list("ftp://mirrors.ibiblio.org/", auth=("anonymous", "[email protected]",))

This fails:

>>> resp = s.list("ftp://mirrors.ibiblio.org", auth=("anonymous", "[email protected]",))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\test\lib\site-packages\requests_ftp\ftp.py", line 21, in list_helper
    return self.request('LIST', url, **kwargs)
  File "c:\test\lib\site-packages\requests\sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\test\lib\site-packages\requests\sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "c:\test\lib\site-packages\requests_ftp\ftp.py", line 145, in send
    host, port, path = self.get_host_and_path_from_url(request)
  File "c:\test\lib\site-packages\requests_ftp\ftp.py", line 288, in get_host_and_path_from_url
    if path[0] == '/':
IndexError: string index out of range

It is probably best to test with startswith rather than a string index or to check if the string exists or else ;)

support reporting back the "last-modified" time in response headers

to stay close(r) to handling of http urls by requests

Cache mechanism for requests-ftp

Hello,

I wonder if requests-ftp could support cache.
It will be great for pydata/pandas-datareader#145
see pydata/pandas-datareader#143

See also requests-cache/requests-cache#57

Kind regards

how to download & upload with this ?

this rep can only send requests to ftp server ?
I can't even further processs with the response, (like anlalize the link recursively ,fetch the file & directory etc)

Missing StringIO import from ftp.py

Import is missing from ftp.py:
from StringIO import StringIO

fetches entire content into memory first even with stream=True? (via BytesIO etc)

didn't dig inside but my requests-based code [1] has failed while I was accessing ftp url and expected the requests.models.Response.raw having .stream method (since apparently for http it provided by requests.packages.urllib3.response.HTTPResponse). With requests_ftp, I just got BytesIO object, which was apparently downloaded in full while establishing the session using FTPSession.get(url, stream=True).
So, streaming of the content is not "supported" (yet)?
FWIW requests_ftp v 0.3.1 installed from pip

[1] if not scared, check it here

0.3.1 + be9af0d7: pytest is failing

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes oly locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra tests
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/requests-ftp-0.3.1
plugins: forked-1.4.0, xdist-2.5.0
collected 19 items

tests/test_ftp.py ............                                                                                                                                       [ 63%]
tests/test_ftp_proxy.py FFFF                                                                                                                                         [ 84%]
tests/unit/test_status_code_interpret.py ...                                                                                                                         [100%]

================================================================================= FAILURES =================================================================================
______________________________________________________________________________ test_proxy_get ______________________________________________________________________________

ftpd = <simple_ftpd.SimpleFTPServer at 0x7f70b9596dc0>, proxy = <simple_proxy.ProxyServer object at 0x7f70b95c42e0>
session = <requests_ftp.ftp.FTPSession object at 0x7f70b958f7c0>

    def test_proxy_get(ftpd, proxy, session):
        # Create a file in the anonymous root and fetch it through a proxy
        with _prepareTestData(ftpd.anon_root) as (testfile, testdata):
            testurl = 'ftp://127.0.0.1:%d/%s' % (ftpd.ftp_port, testfile)
>           response = session.get(testurl, proxies={'ftp': 'localhost:%d' % proxy.port})

tests/test_ftp_proxy.py:36:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/requests/sessions.py:542: in get
    return self.request('GET', url, **kwargs)
/usr/lib/python3.8/site-packages/requests/sessions.py:529: in request
    resp = self.send(prep, **send_kwargs)
/usr/lib/python3.8/site-packages/requests/sessions.py:645: in send
    r = adapter.send(request, **kwargs)
../../BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages/requests_ftp/ftp.py:218: in send
    return self.send_proxy(request, proxy, **kwargs)
../../BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages/requests_ftp/ftp.py:283: in send_proxy
    adapter = s.get_adapter(proxy_url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <requests_ftp.ftp.FTPSession object at 0x7f70b963cfa0>, url = 'localhost:///35693'

    def get_adapter(self, url):
        """
        Returns the appropriate connection adapter for the given URL.

        :rtype: requests.adapters.BaseAdapter
        """
        for (prefix, adapter) in self.adapters.items():

            if url.lower().startswith(prefix.lower()):
                return adapter

        # Nothing matches :-/
>       raise InvalidSchema("No connection adapters were found for {!r}".format(url))
E       requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:///35693'

/usr/lib/python3.8/site-packages/requests/sessions.py:732: InvalidSchema
______________________________________________________________________ test_proxy_connection_refused _______________________________________________________________________

ftpd = <simple_ftpd.SimpleFTPServer at 0x7f70b9596dc0>, session = <requests_ftp.ftp.FTPSession object at 0x7f70b845f6a0>

    def test_proxy_connection_refused(ftpd, session):
        # Create and bind a socket but do not listen to ensure we have a port
        # that will refuse connections
        def target(s, goevent):
            goevent.set()

        with socketServer(target) as port:
            with pytest.raises(requests.exceptions.ConnectionError):
>               session.get(
                    'ftp://127.0.0.1:%d/' % ftpd.ftp_port,
                    proxies={'ftp': 'localhost:%d' % port})

tests/test_ftp_proxy.py:58:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/requests/sessions.py:542: in get
    return self.request('GET', url, **kwargs)
/usr/lib/python3.8/site-packages/requests/sessions.py:529: in request
    resp = self.send(prep, **send_kwargs)
/usr/lib/python3.8/site-packages/requests/sessions.py:645: in send
    r = adapter.send(request, **kwargs)
../../BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages/requests_ftp/ftp.py:218: in send
    return self.send_proxy(request, proxy, **kwargs)
../../BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages/requests_ftp/ftp.py:283: in send_proxy
    adapter = s.get_adapter(proxy_url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <requests_ftp.ftp.FTPSession object at 0x7f70b95d0ca0>, url = 'localhost:///34353'

    def get_adapter(self, url):
        """
        Returns the appropriate connection adapter for the given URL.

        :rtype: requests.adapters.BaseAdapter
        """
        for (prefix, adapter) in self.adapters.items():

            if url.lower().startswith(prefix.lower()):
                return adapter

        # Nothing matches :-/
>       raise InvalidSchema("No connection adapters were found for {!r}".format(url))
E       requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:///34353'

/usr/lib/python3.8/site-packages/requests/sessions.py:732: InvalidSchema
_________________________________________________________________________ test_proxy_read_timeout __________________________________________________________________________

ftpd = <simple_ftpd.SimpleFTPServer at 0x7f70b9596dc0>, session = <requests_ftp.ftp.FTPSession object at 0x7f70b9584a00>

    def test_proxy_read_timeout(ftpd, session):
        # Create and accept a socket, but never respond
        def target(s, goevent, event):
            s.listen(1)
            goevent.set()
            (clientsock, _addr) = s.accept()
            try:
                event.wait(5)
            finally:
                clientsock.close()

        event = threading.Event()
        with socketServer(target, event) as port:
            with pytest.raises(requests.exceptions.ReadTimeout):
>               session.get(
                    'ftp://127.0.0.1:%d' % ftpd.ftp_port,
                    proxies={'ftp': 'localhost:%d' % port},
                    timeout=1)

tests/test_ftp_proxy.py:77:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/requests/sessions.py:542: in get
    return self.request('GET', url, **kwargs)
/usr/lib/python3.8/site-packages/requests/sessions.py:529: in request
    resp = self.send(prep, **send_kwargs)
/usr/lib/python3.8/site-packages/requests/sessions.py:645: in send
    r = adapter.send(request, **kwargs)
../../BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages/requests_ftp/ftp.py:218: in send
    return self.send_proxy(request, proxy, **kwargs)
../../BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages/requests_ftp/ftp.py:283: in send_proxy
    adapter = s.get_adapter(proxy_url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <requests_ftp.ftp.FTPSession object at 0x7f70b9621f10>, url = 'localhost:///45841'

    def get_adapter(self, url):
        """
        Returns the appropriate connection adapter for the given URL.

        :rtype: requests.adapters.BaseAdapter
        """
        for (prefix, adapter) in self.adapters.items():

            if url.lower().startswith(prefix.lower()):
                return adapter

        # Nothing matches :-/
>       raise InvalidSchema("No connection adapters were found for {!r}".format(url))
E       requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:///45841'

/usr/lib/python3.8/site-packages/requests/sessions.py:732: InvalidSchema
_______________________________________________________________________ test_proxy_connection_close ________________________________________________________________________

ftpd = <simple_ftpd.SimpleFTPServer at 0x7f70b9596dc0>, session = <requests_ftp.ftp.FTPSession object at 0x7f70b958bca0>

    def test_proxy_connection_close(ftpd, session):
        # Create and accept a socket, then close it
        def target(s, goevent):
            s.listen(1)
            goevent.set()
            (clientsock, _addr) = s.accept()
            clientsock.close()

        with socketServer(target) as port:
            with pytest.raises(requests.exceptions.ConnectionError):
>               session.get(
                    'ftp://127.0.0.1:%d/' % ftpd.ftp_port,
                    proxies={'ftp': 'localhost:%d' % port},
                    timeout=1)

tests/test_ftp_proxy.py:93:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/requests/sessions.py:542: in get
    return self.request('GET', url, **kwargs)
/usr/lib/python3.8/site-packages/requests/sessions.py:529: in request
    resp = self.send(prep, **send_kwargs)
/usr/lib/python3.8/site-packages/requests/sessions.py:645: in send
    r = adapter.send(request, **kwargs)
../../BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages/requests_ftp/ftp.py:218: in send
    return self.send_proxy(request, proxy, **kwargs)
../../BUILDROOT/python-requests-ftp-0.3.1-24.fc35.x86_64/usr/lib/python3.8/site-packages/requests_ftp/ftp.py:283: in send_proxy
    adapter = s.get_adapter(proxy_url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <requests_ftp.ftp.FTPSession object at 0x7f70b95e62e0>, url = 'localhost:///60247'

    def get_adapter(self, url):
        """
        Returns the appropriate connection adapter for the given URL.

        :rtype: requests.adapters.BaseAdapter
        """
        for (prefix, adapter) in self.adapters.items():

            if url.lower().startswith(prefix.lower()):
                return adapter

        # Nothing matches :-/
>       raise InvalidSchema("No connection adapters were found for {!r}".format(url))
E       requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:///60247'

/usr/lib/python3.8/site-packages/requests/sessions.py:732: InvalidSchema
========================================================================= short test summary info ==========================================================================
FAILED tests/test_ftp_proxy.py::test_proxy_get - requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:///35693'
FAILED tests/test_ftp_proxy.py::test_proxy_connection_refused - requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:///34353'
FAILED tests/test_ftp_proxy.py::test_proxy_read_timeout - requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:///45841'
FAILED tests/test_ftp_proxy.py::test_proxy_connection_close - requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:///60247'
======================================================================= 4 failed, 15 passed in 2.40s =======================================================================

PyPI link to repo wrong

https://pypi.python.org/pypi/requests-ftp/0.2.0

has

http://github.com/Lukasa/requests_ftp

in the metadata ("Homepage" part). s/_/-/ :).

Also may be good to include a correct link under Contribute section.

EDIT: Er, you have fixed it in setup.py. Maybe it needs a new release in order to update it. Feel free to close if so.

demo is errro

Requests version in requirements.txt is too old, change to >= 2.3.0

Requests 1.0.0+ is listed in requirements.txt, but Requests FTP relies on ConnectTimeout which only appeared as of c2aeaa3, probably Requests version 2.3.0:

  File "/usr/local/lib/python2.7/dist-packages/requests_ftp/__init__.py", line 21, in <module>
    from .ftp import FTPAdapter, monkeypatch_session
  File "/usr/local/lib/python2.7/dist-packages/requests_ftp/ftp.py", line 13, in <module>
    from requests.exceptions import ConnectionError, ConnectTimeout, ReadTimeout
ImportError: cannot import name ConnectTimeout

response codes are of the type string

Which fails a code verification that is common in requests of the type

if r.status_code == 226: myprocess

KeyError: 'GET'

I am trying to use this at conda/conda#682 to get FTP support for downloading. I am getting

 File "/Users/aaronmeurer/Documents/Continuum/conda/conda/connection.py", line 172, in send
    resp = self.func_table[request.method](path, request)
KeyError: 'GET'

(that corresponds to https://github.com/Lukasa/requests-ftp/blob/master/requests_ftp/ftp.py#L157).

Anyway way to support GET? Do I just need to mount this differently? I am currently doing self.mount("ftp://", FTPAdapter()) in my Session subclass.

Enable continuous integration with Travis

Hello,

it will be nice if Continuous Integration (with Travis for example) could be enabled.

Kind regards

pyfilesystem sftp filesystem interface

http://pythonhosted.org/fs/filesystems.html#sftp-secure-ftp

http://pythonhosted.org/fs/interface.html

http://pythonhosted.org/fs/implementersguide.html#essential-methods