hpchrist / mountpoint-s3

A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.

Mountpoint for Amazon S3

Mountpoint for Amazon S3 is a simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system. With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like open and read. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface.

File system features: Mountpoint for Amazon S3 is optimized for read-heavy workloads that need high throughput. It does not implement the full POSIX specification for file systems, and we don't intend to implement features that cannot be efficiently executed against S3's object APIs. See SEMANTICS.md for a detailed description of Mountpoint for Amazon S3's behavior and POSIX support. For workloads that need POSIX support, we recommend AWS fully managed file system services like Amazon FSx for Lustre and its support for linking S3 buckets.

Compatibility: Mountpoint for Amazon S3 is designed for high-performance access to the Amazon S3 service. While it may be functional against other storage services that use S3-like APIs, we aren't able to provide support for those use cases, and they may inadvertently break when we make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance improvements for these services if the changes can be tested against Amazon S3.

Current status

Mountpoint for Amazon S3 is currently an alpha release and should not be used in production. We're tracking its production readiness and future features on the Mountpoint for Amazon S3 public roadmap. We're especially interested in early feedback on features, performance, and compatibility. Please send feedback by opening a GitHub issue.

There are some notable restrictions in this first release:

• Mountpoint for Amazon S3 is currently read-only, so you won't be able to write objects back to S3 through the file system. We're working on allowing sequential writes to new objects in a future release.
• Mountpoint for Amazon S3 does not cache any object data or metadata.
• The only way to install the alpha release of Mountpoint for Amazon S3 is by compiling from source (see Getting started below). This will change in a future release.
• Manual endpoint configuration might be required for some S3 customers (see Configuration below).

Getting started

The alpha release of Mountpoint for Amazon S3 is only available by building from source, and only supports Linux.

First, install the necessary dependencies. For RPM-based distributions (Amazon Linux 2 and 2023, Fedora, CentOS, etc):

sudo yum install fuse fuse-devel cmake3 clang-devel git

or for apt-based distributions (Debian, Ubuntu, etc):

sudo apt install fuse libfuse-dev cmake libclang-dev git

Secondly, install the Rust compiler via rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

Now clone this repository and its submodules:

git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git

Finally, compile the client:

cd mountpoint-s3
cargo build --release

The final binary will be at target/release/mount-s3.

The following instructions assume you have the binary on your path. Add it to a location on your path now, or prefix mount-s3 commands with the path to the binary.

To use the client, ensure you have access to valid AWS credentials. For example, you could create a new IAM user and add it to the ~/.aws/credentials file, or configure an EC2 instance with an IAM role.

Then run the client, specifying the directory in which your S3 bucket should be mounted (here, ~/mnt):

mkdir ~/mnt
mount-s3 my-s3-bucket-name ~/mnt

The client will run in the background by default, and the ~/mnt directory now gives access to the objects in your S3 bucket.

Mountpoint for Amazon S3 using Docker

First, build the docker image using the below command.

cd mountpoint-s3/docker
docker build -t mount-s3 .

Once the container build is successful, configure AWS credentials as environment variables. You can set short-term or long-term credentials using variables, such as those in the AWS CLI environment variables documentation. Run the following command to mount the S3 bucket and list some files. Replace DOC-EXAMPLE-BUCKET with the name of your bucket.

 docker run -it \
    --cap-add SYS_ADMIN \
    --device /dev/fuse \
    --env AWS_ACCESS_KEY_ID \
    --env AWS_SECRET_ACCESS_KEY \
    --env AWS_SESSION_TOKEN \
    --env AWS_DEFAULT_REGION \
    mount-s3 /bin/bash -c "./mount-s3 DOC-EXAMPLE-BUCKET /mnt; ls -l /mnt"

Mountpoint for Amazon S3 using AUR (Unofficial)

If you're using Arch Linux, you can use the unofficial AUR package. Using Yay:

yay -S mountpoint-s3-git

Configuration

For the alpha release, additional configuration is required to use Mountpoint for Amazon S3 in some circumstances:

• Multiple AWS CLI profiles: set the AWS_PROFILE environment variable to the profile you want to use. Profile selection will be supported via the --profile command-line argument in the future (awslabs#151).
• High-bandwidth EC2 instances: manually specify the available network bandwidth in Gbps with the --throughput-target-gbps command-line argument. Defaults to 10 Gbps if not configured. This configuration will be automated in the future (awslabs#3).
• AWS GovCloud, China (Beijing), and China (Ningxia) regions: manually specify the S3 endpoint with the --endpoint-url command-line argument and the region name with the --region argument. For example, for the AWS GovCloud (US-West) region, specify --endpoint-url https://s3.us-gov-west-1.amazonaws.com --region us-gov-west-1. See the S3 endpoints documentation for more details. These regions will be supported via only the --region command-line argument in the future (awslabs#4).
• Access points: use the bucket-style access point alias for the access point as the bucket name to mount.
• FIPS, dualstack, and transfer acceleration endpoints: manually specify the S3 endpoint with the --endpoint-url command-line argument. See the S3 endpoints documentation for more details. These endpoints will be configurable in the future (awslabs#4).

Contributing

We welcome contributions to Mountpoint for Amazon S3! Please see CONTRIBUTING.md for more information on how to report bugs or submit pull requests, LOGGING.md for details on how to capture logging data for bug reports, and BENCHMARKING.md for performance regression testing.

Security

If you discover a potential security issue in this project we ask that you notify AWS Security via our vulnerability reporting page. Please do not create a public GitHub issue.

Code of conduct

This project has adopted the Amazon Open Source Code of Conduct. See CODE_OF_CONDUCT.md for more details.

License

This project is licensed under the Apache-2.0 License. It builds on a number of other awesome projects with open source licenses, primarily:

• The fuser bindings for FUSE (MIT license)
• The AWS Common Runtime for interacting with S3 (Apache-2.0 license)

See deny.toml for a list of licenses used by our dependencies.