bolcom / libunftp

Extensible, async, cloud orientated FTP(S) server library and the core of unFTP: https://github.com/bolcom/unFTP. Follow up and talk to us on https://t.me/unftp

libunftp

When you need to FTP, but don't want to.

Website | API Docs | **unFTP **

The libunftp library drives unFTP. It's an extensible, async, cloud orientated FTP(S) server implementation in Rust brought to you by the bol.com techlab.

Because of its plug-able authentication (e.g. PAM, JSON File, Generic REST) and storage backends (e.g. local filesystem, Google Cloud Storage) it's more flexible than traditional FTP servers and a perfect match for the cloud.

It runs on top of the Tokio asynchronous run-time and tries to make use of Async IO as much as possible.

Feature highlights:

• 39 Supported FTP commands (see commands directory) and growing
• Ability to implement own storage back-ends
• Ability to implement own authentication back-ends
• Explicit FTPS (TLS)
• Mutual TLS (Client certificates)
• TLS session resumption
• Prometheus integration
• Structured Logging
• Proxy Protocol support
• Automatic session timeouts
• Per user IP allow lists

Known storage back-ends:

• unftp-sbe-fs - Stores files on the local filesystem
• unftp-sbe-gcs - Stores files in Google Cloud Storage
• unftp-sbe-rooter - Wraps another storage back-end in order to root a user to a specific home directory.
• unftp-sbe-restrict - Wraps another storage back-end in order to restrict the FTP operations a user can do i.e. provide authorization.

Known authentication back-ends:

• unftp-auth-jsonfile - Authenticates against JSON text.
• unftp-auth-pam - Authenticates via PAM.
• unftp-auth-rest - Consumes an HTTP API to authenticate.

Prerequisites

You'll need Rust 1.41 or higher to build libunftp.

Getting started

If you've got Rust and cargo installed, create your project with

cargo new myftp

Add the libunftp and tokio crates to your project's dependencies in Cargo.toml. Then also choose a storage back-end implementation to add. Here we choose the file system back-end:

[dependencies]
libunftp = "0.19.0"
unftp-sbe-fs = "0.2"
tokio = { version = "1", features = ["full"] }

Now you're ready to develop your server! Add the following to src/main.rs:

use unftp_sbe_fs::ServerExt;

#[tokio::main]
pub async fn main() {
    let ftp_home = std::env::temp_dir();
    let server = libunftp::Server::with_fs(ftp_home)
        .greeting("Welcome to my FTP server")
        .passive_ports(50000..65535)
        .build()
        .unwrap();

    server.listen("127.0.0.1:2121").await;
}

You can now run your server with cargo run and connect to localhost:2121 with your favourite FTP client e.g.:

lftp -p 2121 localhost

For more help refer to:

• the examples directory.
• the API Documentation.
• unFTP server, a server from the bol.com techlab that is built on top of libunftp.
• this blog post about libunftp and unFTP.

Getting help and staying informed

Support is given on a best effort basis. You are welcome to engage us on the discussions page or create a Github issue.

You can also follow news and talk to us on Telegram

Contributing

Thank you for your interest in contributing to libunftp!

Please feel free to create a Github issue if you encounter any problems.

Want to submit a feature request or develop your own storage or authentication back-end? Then head over to our contribution guide (CONTRIBUTING.md).

License

You're free to use, modify and distribute this software under the terms of the Apache License v2.0.

libunftp's Issues

Fix Data Channel abort in TLS mode (Filezilla test)

When connecting to unFTP with FileZilla we get the following error:

This is probably related to something we've seen before when testing with lftp where we close the stream to early.

Implement REST in the GCS backend (upload)

Migrate to std::future::Future and async/await

In anticipation of async/await coming to stable Rust on November 7th, libraries like Tokio and Hyper have been preparing support for std::future::Future and eventually plan on dropping futures 0.1 entirely. It would be cool if libunftp came prepared for these major improvements in ergonomics too!

Add check if connecting client on data port is the same as the one initiating the ftp transfer

Since we're really low on ports when running in kubernetes, we need a way to limit attackers connecting randomly to data ports. While this feature breaks FTP protocol compatbility, it is necessary for security under certan circumstances.

Stuck "waiting for welcome message" (Filezilla test)

Ocassionally I see this when testing (lib)unFTP with FileZilla:

Steps to reproduce:

Its not very consistent but here goes:

1. Start unFTP with certificates (TLS) configured:

cargo run -- \
--bind-address=0.0.0.0:2122 \
--bind-address-http=0.0.0.0:8080 \
--passive-ports=50000-65535 \
--ftps-certs-file=/Users/xxx/unftp/unftp.crt
--ftps-key-file=/Users/xxx/unftp/unftp.key
--root-dir=/Users/xxx/unftp/home

2. Start FileZilla and connect to unFTP:

Expected Result:

FileZilla should be able to log in immediately:

Status:      	Resolving address of localhost
Status:      	Connecting to [::1]:2122...
Status:      	Connection attempt failed with "ECONNREFUSED - Connection refused by server", trying next address.
Status:      	Connecting to 127.0.0.1:2122...
Status:      	Connection established, waiting for welcome message...
Status:      	Initializing TLS...
Status:      	Verifying certificate...
Status:      	TLS connection established.
Status:      	Logged in
Status:      	Retrieving directory listing of "/"...
Status:      	Calculating timezone offset of server...
Status:      	Timezone offset of server is 0 seconds.
Status:      	Directory listing of "/" successful

Actual Result:

FileZilla times out after 20 seconds and does a couple of retries but with no avail.

Fix Builds

Get rid of std::sync::Mutex, since it can deadlock unFTP

Add single-data port support

While deploying to kubernetes, we realized we don't actually need so many ports open in kubernetes if we use a single data port only.

(details: https://serverfault.com/questions/270745/why-does-ftp-passive-mode-require-a-port-range-as-opposed-to-only-one-port)

The upside of using a single port for control channel and another single port for data are many:

• kubernetes has no support for port ranges, which makes opening up thousands of ports a very remote possibility only;
• opening up a lot of ports is always seen as an impact on management, configuration and security in general;
• by identifying clients on the data port based on the control port's IP, we get added security against hijack attacks.

There are also downsides:

• There could be weird use cases where the data port is opened from another IP than the control port. This is normally reserved for advanced FTP usage that is not at all common these days.

• If a client connects multiple times from the same IP, and they both initiate data transfer, we can't identify which data port connection belongs to which client. A possible workaround would be

rename command always reports success

rename command is spawned, errors are not returned the client

Implement the MDTM command (rfc3659)

Implement MDTM. See https://tools.ietf.org/html/rfc3659#page-8

Randomize data port on PASV

Implement error handling for put

Data port connections allowed from control port's IP only

While this can cause break in some rare ftp users, for 99% of the cases, it's a good security measure to forego port scanning for recently opened ftp data ports (and this snooping data).

Implement error handling for mkd

Update directory timestamps if a file changes inside the directory

Leaking user and group IDs in filesystem backend mode

libunftp shows the actual owner uid and gid in the LIST output. I think this is a minor security issue. It should instead just return user ftp and group ftp.

We should make this configurable. vsftpd has a hide_ids option that is set to YES by default.

The question is, are we even bothered by this since we are using containers? Guess not?

Storage back end should propagate the futures that they implement

Since storage back ends can have features not implemented (e.g.: SIZE), but some clients change behavior based on the available features, so the back ends should be able to tell what features they support.

Implement RENAME in the GCS backend

Currently unimplemented. See here:

libunftp/crates/unftp-sbe-gcs/src/lib.rs

Line 358 in b53478b

Err(Error::from(ErrorKind::CommandNotImplemented))

Fix the AsAsyncReads for the Cloud storage

The cloud storage back end is not working since AsAsyncReads is unimplemented. Since the library is now only dependent on tokio02 and futures03, it might be that this trait is not needed anymore.

Support FileZilla with the Google Cloud Storage backend

Since the Google Cloud Storage backend doesn't support directory times, the output of a LIST is like this:

drwxr-xr-x            0            0              0            - mydirectory
drwxr-xr-x            0            0              0            - test
drwxr-xr-x            0            0              0 Dec 13 14:52 /
-rwxr-xr-x            0            0             24 Jun 07 13:05 test_thingy
-rwxr-xr-x            0            0             21 Jun 07 13:06 thingy

FileZilla thinks that the directories are - /, - mydirectory, - test instead of /, mydirectory, test

Unftp hangs when authentication fails.

Do not block on the command channel

Do not use .wait() while handling the commands

Clean up error handling

There are lot of unwraps in the code, they should be removed, or documented why unwrap is the way to go. This is an umbrella task: Pull requests are probably best to send per file to keep it small.

Implement REST command (RFC 3659)

See https://tools.ietf.org/html/rfc3659#page-13

Check for the http status in the GCS storage and propagate the error according to that

The GCS backend is ignoring the http status code and error body in most of the commands. These information should be used for error propagation. The 'del' command can be used as an example how to do it.

Upgrade Futures, Tokio and Hyper

Upgrade to Futures 0.3, Tokio 0.2 and Hyper 0.13

Implement error handling for metadata

Implement REST in the GCS backend (download)

Currently unimplemented.

Add structured logging

At the moment, libunftp has a few, scattered, built-in log-based logging.
For a proper production environment, we need the possibility of structured logging (via slog?), or an event system that allows actual logging implementations (or anything else) on the user of the library.

Sometimes parameters seem to be not recognized as filenames

When giving a nonexistent filename like t, f, unftp does not display the filename in the error. It looks like some such characters are treated as if they are not a filename within libunftp. E.g.:

lftp localhost:/> rm t
rm: Access failed: 550 File not found
lftp localhost:/> rm u
rm: Access failed: 550 File not found
lftp localhost:/> rm d
rm: Access failed: 550 File not found
lftp localhost:/> rm e
rm: Access failed: 550 File not found
lftp localhost:/> rm f
rm: Access failed: 550 File not found

Some other characters don't have the issue:

lftp localhost:/> rm s
rm: Access failed: 550 File not found (s)
lftp localhost:/> rm v
rm: Access failed: 550 File not found (v)
lftp localhost:/> rm w
rm: Access failed: 550 File not found (w)

Although this looks like a benign bug, it could be more than that.

Implement the CDC command

GCS backend error handling

The GCS backend should have a generic implementation for handling error responses from the GCS API, safely and consistently. It should also make troubleshooting easy through verbose error logging.

The main concern here is hitting API request count limits, caused by client retries: (causing 429—Too Many Requests)

1. Error responses should be consistent between commands (permanent vs temporary failures). Retryable error codes should only be send when there is an actual transient error.
2. Should implement exponential backoff. There is a separate issue for that: #82

Troubleshooting: It should verbosely log these errors, and it should include details retrieved through the HTTP response body (JSON). See GCS status codes.

Upgrade futures in auth/jsonfile_auth

Implement SIZE in the GCS backend

Currently unimplemented.

Implement the EPSV command

"The EPSV command requests that a server listen on a data port and wait for a connection."

Spec: https://tools.ietf.org/html/rfc2428#section-3

See also https://www.jscape.com/blog/what-is-the-ftp/s-epsv-command-and-when-do-you-use-it

`U` can only use `libunftp::auth::AnonymousUser`

Thank you as always :-))

Please tell us about U. I want to give my own user.

use futures::{future, Future, Stream};
use std::path::{Path, PathBuf};
use std::time::SystemTime;

#[derive(Clone, Debug)]
struct SampleUser {
    hash: String,
}

impl AsRef<SampleUser> for SampleUser {
    fn as_ref(&self) -> &SampleUser {
        self
    }
}

struct SampleStorage;

struct SampleMetadata;

impl libunftp::storage::Metadata for SampleMetadata {
    fn len(&self) -> u64 {
        0
    }

    fn is_empty(&self) -> bool {
        self.len() == 0
    }

    fn is_dir(&self) -> bool {
        true
    }

    fn is_file(&self) -> bool {
        !self.is_dir()
    }

    fn is_symlink(&self) -> bool {
        false
    }

    fn modified(&self) -> Result<SystemTime, libunftp::storage::Error> {
        Ok(SystemTime::now())
    }

    fn gid(&self) -> u32 {
        48
    }

    fn uid(&self) -> u32 {
        48
    }
}

impl libunftp::storage::StorageBackend<SampleUser> for SampleStorage {
    type File = std::io::Cursor<Vec<u8>>;
    type Metadata = SampleMetadata;
    type Error = libunftp::storage::Error;

    fn stat<P: AsRef<Path>>(
        &self,
        _user: &Option<SampleUser>,
        _path: P,
    ) -> Box<dyn Future<Item = Self::Metadata, Error = Self::Error> + Send> {
        Box::new(future::ok(Self::Metadata {}))
    }

    fn list<P: AsRef<Path>>(
        &self,
        _user: &Option<SampleUser>,
        _path: P,
    ) -> Box<
        dyn Stream<
                Item = libunftp::storage::Fileinfo<std::path::PathBuf, Self::Metadata>,
                Error = Self::Error,
            > + Send,
    >
    where
        <Self as libunftp::storage::StorageBackend<SampleUser>>::Metadata: libunftp::storage::Metadata,
    {
        let result = future::ok(vec!["a", "b", "c"])
            .map(|_| {
                futures::stream::iter_ok(vec![libunftp::storage::Fileinfo {
                    path: PathBuf::from("".to_string()),
                    metadata: Self::Metadata {},
                }])
            })
            .flatten_stream();

        Box::new(result)
    }

    fn get<P: AsRef<Path>>(
        &self,
        _user: &Option<SampleUser>,
        _path: P,
    ) -> Box<dyn Future<Item = std::io::Cursor<Vec<u8>>, Error = Self::Error> + Send> {
        Box::new(future::ok(std::io::Cursor::new(vec![0])))
    }

    fn put<P: AsRef<Path>, R: tokio::prelude::AsyncRead + Send + 'static>(
        &self,
        _user: &Option<SampleUser>,
        _bytes: R,
        _path: P,
    ) -> Box<dyn Future<Item = u64, Error = Self::Error> + Send> {
        Box::new(future::ok(0))
    }

    fn del<P: AsRef<Path>>(
        &self,
        _user: &Option<SampleUser>,
        _path: P,
    ) -> Box<dyn Future<Item = (), Error = std::io::Error> + Send> {
        Box::new(future::ok(()))
    }

    fn rmd<P: AsRef<Path>>(
        &self,
        _user: &Option<SampleUser>,
        _path: P,
    ) -> Box<dyn Future<Item = (), Error = Self::Error> + Send> {
        Box::new(future::ok(()))
    }

    fn mkd<P: AsRef<Path>>(
        &self,
        _user: &Option<SampleUser>,
        _path: P,
    ) -> Box<dyn Future<Item = (), Error = Self::Error> + Send> {
        Box::new(future::ok(()))
    }

    fn rename<P: AsRef<Path>>(
        &self,
        _user: &Option<SampleUser>,
        _from: P,
        _to: P,
    ) -> Box<dyn Future<Item = (), Error = Self::Error> + Send> {
        Box::new(future::ok(()))
    }
}

fn main() {
    let mut rt = tokio::runtime::Runtime::new().unwrap();
    let server = libunftp::Server::new(Box::new(move || { SampleStorage {} }));
    rt.spawn(server.listener("127.0.0.1:2121"));
}

However, an error occurs :'-(

   Compiling ftp-sample v0.1.0 (/home/kenta/dev/ninja/ftp-sample)
error[E0277]: the trait bound `SampleStorage: libunftp::storage::StorageBackend<libunftp::auth::AnonymousUser>` is not satisfied
   --> src/main.rs:145:18
    |
145 |     let server = libunftp::Server::new(Box::new(move || { SampleStorage {} }));
    |                  ^^^^^^^^^^^^^^^^^^^^^ the trait `libunftp::storage::StorageBackend<libunftp::auth::AnonymousUser>` is not implemented for `SampleStorage`
    |
    = help: the following implementations were found:
              <SampleStorage as libunftp::storage::StorageBackend<SampleUser>>
    = note: required by `libunftp::server::Server::<S, U>::new`

This is successfly :)

+ impl libunftp::storage::StorageBackend<libunftp::auth::AnonymousUser> for SampleStorage {
- impl libunftp::storage::StorageBackend<SampleUser> for SampleStorage {

Currently, U is only libunftp::auth::AnonymousUser can be passed. I wrote a code for libunftp to compile successfully here.

Is there a good solution?

thank you.

Implement error handling for list

Filezilla thinks unFTP doesn't support UTF-8

When connecting to unFTP from FileZilla this is the result:

Expected:

The message: "Status: Server does not support non-ASCII characters" must not appear

multibyte string error

Hello! I facing a problem.

Success case

Command:	PASV
Response:	227 Entering Passive Mode (127,0,0,1,234,79)
Command:	STOR a
Response:	150 Ready to receive data
Response:	226 File successfully written
Status:	File transfer successful, transferred 0 B in 1 second

Error case

Command:	PASV
Response:	227 Entering Passive Mode (127,0,0,1,203,51)
Command:	STOR あ
Response:	500 Invalid UTF8 in command
Error:	Critical file transfer error

Error from here
https://github.com/bolcom/libunftp/blob/master/src/server/commands/mod.rs#L593-L595

How are you working it out?

Make HELP command useful

The help command currently doesn't do anything useful.

When you connect with and FTP client to unFTP and issue the HELP command it currently just responds with "powered by unFTP". Perhaps it can also give a link to our website (https://unftp.rs)

Implement RMD in the GCS backend

Currently unimplemented.

"ls <partial filename>" works, while it should not have 'pattern match' behavior

This behavior is not correct:

---> LIST t
<--- 150 Sending directory list
---- Got EOF on data connection
---- Closing data socket
-rwxr-xr-x            0            0             24 Jun 07 13:05 test_thingy
-rwxr-xr-x            0            0             21 Jun 07 13:06 thingy
<--- 226 Listed the directory 

expected instead:

550 File or directory not found

libunftp doesn't seem to work with macOS Finder's integrated FTP

When running libunftp via unFTP Server and then trying to connect with macOS Finder's integrated FTP I get an error message:

I run unFTP Server with:

cargo run -- \
  --bind-address=0.0.0.0:2121 \
  --bind-address-http=0.0.0.0:8080 \
  --home-dir=/Users/xxx/Desktop/unftp/home \

This is what I see in the logs (Note: The double logging is probably an issue with the log setup and not macOS's sending double commands)

 Oct 04 09:09:10.340 INFO Starting unFTP server., sbe-type: filesystem, auth-type: anonymous, home: /Users/hdejager/Desktop/unftp/home, address: 0.0.0.0:2121, version: v0.3.0-5-ge46b146
 Oct 04 09:09:10.341 INFO Starting Prometheus unFTP exporter., address: 0.0.0.0:8080
Oct 04 09:09:10.343 INFO Using anonymous authenticator
Oct 04 09:09:21.100 INFO Processing event Command(User { username: b"anonymous" })
Oct 04 09:09:21.100 INFO Processing event Command(User { username: b"anonymous" })
Oct 04 09:09:21.100 INFO Processing event Command(Pass { password: b"[email protected]" })
Oct 04 09:09:21.100 INFO Processing event Command(Pass { password: b"[email protected]" })
Oct 04 09:09:21.100 INFO Processing event InternalMsg(AuthSuccess)
Oct 04 09:09:21.100 INFO Processing event InternalMsg(AuthSuccess)
Oct 04 09:09:21.101 INFO Processing event Command(Syst)
Oct 04 09:09:21.101 INFO Processing event Command(Syst)
Oct 04 09:09:21.101 INFO Processing event Command(Pwd)
Oct 04 09:09:21.101 INFO Processing event Command(Pwd)
Oct 04 09:09:21.101 INFO Processing event Command(Type)
Oct 04 09:09:21.101 INFO Processing event Command(Type)
Oct 04 09:09:21.102 INFO Processing event Command(Cwd { path: "/" })
Oct 04 09:09:21.102 INFO Processing event Command(Cwd { path: "/" })
Oct 04 09:09:21.102 INFO Processing event Command(Port)
Oct 04 09:09:21.102 INFO Processing event Command(Pasv)

Redesign

Hi guys.

I thinks processing TCP of this library is complicated.
The reason is control connection and data connection interact in both directions by channel.
I redesigned the process in here (in progress).

My idia will all comannds process in process() and responde(). Session::process_data() and channels be unnecessarily.

How do you think that?

GCS backend implement exponential backoff

To avoid hitting API request count limits, caused by client retries: (causing 429—Too Many Requests)
Implement exponential backoff.

Implement the APPE command

"This command causes the server-DTP to accept the data transferred via the data connection and to store the data in a file at the server site. If the file specified in the pathname exists at the server site, then the data shall be appended to that file; otherwise the file specified in the pathname shall be created at the server site."

Spec: https://www.freesoft.org/CIE/RFC/959/23.htm

Retrieving a non-existent file doesn't result in an error

Split src/auth/mod.rs

mod.rs files should only collect the content of other files and not have any "real" code.

The LIST command returns hardcoded permissions

the LIST command always returns permissions as

rwxr-xr-x

regardless of the actual permissions of the file.

list directories in the cloud storage backend