nvfuse / nvfuse

NVMe based File System in User-space

NVFUSE

NVFUSE is an embeddable file system as a library running in the user-space incorporated with SPDK library which is a user-space NVMe driver newly introduced by Intel. Using this file system applications can directly submit I/O requests to NVMe SSDs. Then, it provides high performance, high reliability as well as POSIX-like interfaces (e.g., nvfuse_open, nvfuse_read, nvfuse_write, and nvfuse_close functions). Note that this file system doesn't make use of well-known FUSE (File System in Userspace) library to provide POSIX compatibility. Key features are listed as follows.

• Running in userspace without any interactions with kernel driver so that interrupts, context switches, and memory copies are eliminated.
• Embeddable file system as a library that introduces new interfaces are similar to POSIX API.
• Simple file system layout that is identical to EXT family file system (e.g., EXT3).
• Strong file system consistency and durability without pessimistic journaling mechanism (e.g., EXT4 journaling) through NVMe metadata feature as described NVMe spec.

Prerequisites

In order to build NVFUSE, some relevant libraries must be configured.

DPDK and SPDK are required.

1) Move onto https://github.com/spdk/spdk
2) DPDK and SPDK must be installed based on their install guide explained in README.

Ubuntu/Debian:

# sudo apt-get install -y gcc make libaio-dev libfuse-dev
# sudo apt-get install -y gcc g++ libcunit1-dev libaio-dev libssl-dev

SPDK compile:

# git clone https://github.com/spdk/spdk.git
# cd ~/spdk
# git submodule update --init
# ./configure
# make

Building

Once the prerequisites are installed, run 'make' within the NVFUSE directory to build the NVFUSE library and examples.

# make SPDK_ROOT_DIR=/home/ysoh/spdk DPDK_DIR=/home/ysoh/spdk/dpdk/build

Before conducting an NVFUSE application, the kernel NVMe driver must be unloaded and some hugepages must be allocated. It can be done with the automation script like the below command line.

# sudo scripts/setup.sh config

Disabling ASLR (Address Space Layout Randomization) is also required like this.

# sudo scripts/disable_aslr.sh

Move to examples directory and execute helloworld.

# cd examples/helloworld
# sudo ./start_helloworld.sh

Make use of performance benchmark. # cd examples/perf # sudo ./start_perf.sh

After finalizing an NVFUSE application, the kernel NVMe driver and hugepages can be reverted with the automation script.

# sudo scripts/setup.sh reset

Examples Not Working (To Be Enabled)
control_plane_proc
dpdk_simple_mp
fio_plugin
libfuse
multi_process
raw_block_device_perf
regression_test
create_1m_files
fio-nvfuse
fsync_test
kernel_filesystem
mkfs
reactor

To do

• File system consistency management via light-weight journaling
• Integration with scalable file system such as Ceph and GlusterFS

Contact

Yongseok Oh ([email protected])

nvfuse's Issues

Compile xattr_test error

Hello:

I got compile error when do making like this:

make SPDK_ROOT_DIR=/home/code/spdk DPDK_DIR=/home/code/spdk/dpdk/build

make -C examples/xattr_test
make: *** examples/xattr_test: No such file or directory. Stop.
make: *** [xattr_test] Error 2

Issue while doing ./mkfs.nvfuse -f

Hello @yongseokoh,

I am working on the nvfuse you have provided. Thanks for this 1st of all.

I am facing few issues while doing mkfs. Can you please help me here..

spdk and dpdk init happens successfully and I can see

 Formatting NVFUSE ...
 Warning: your data will be removed permanently...
--------------------Option------------------------------------------
 nvme format: skipped
 spdk: nvfuse_io_manager = 0x11ff8053b8
 nvfuse_io_manager = 0x11ff8053b8
 sectors = 1875385008, blocks = 234423126
 bg size = 128MB
 num bgs = 7154

 inode size = 128 bytes
 bd_bg_start = 1
 bd_ibitmap_start = 2
 bd_ibitmap_size = 1 blocks
 bd_dbitmap_start = 3
 bd_dbitmap_size = 1 blocks
 itable start = 4
 itable size = 512 blocks
 dtable start = 516
 dtable size = 32252 blocks
 bd end = 32768

There are after I start seeing below errors and abort

nvme_qpair.c: 283:nvme_qpair_print_completion: *NOTICE*: INVALID OPCODE (00/01) sqid:1 cid:254 cdw0:0 sqhd:00d5 p:0 m:0 dnr:0
nvme_qpair.c: 131:nvme_io_qpair_print_command: *NOTICE*: WRITE sqid:1 cid:254 nsid:1 lba:1875116056 len:8
nvme_qpair.c: 283:nvme_qpair_print_completion: *NOTICE*: INVALID OPCODE (00/01) sqid:1 cid:254 cdw0:0 sqhd:00d6 p:0 m:0 dnr:0
mkfs.nvfuse: nvfuse_mkfs.c:53: nvfuse_alloc_root_inode_direct: Assertion `bd->bd_id == bg_id' failed.
Aborted

Thanks in advance