oxfs / oxfs

✨A dead simple, fast SFTP file system ✨

oxfs

A dead simple, fast SFTP file system

Oxfs is a user-space network file system similar to SSHFS, and the underlying data transfer is based on the SFTP protocol. Oxfs introduces an asynchronous refresh policy to solve the jamming problem caused by the mismatch between network speed and user operation file speed. When Oxfs writes a file, it first writes to the local cache file and submits an asynchronous update task to update the content to the remote host. Similarly, when reading a file, it is preferred to read from a local cache file. Oxfs's data cache eventually falls to disk, and even if it is remounted, the history cache can still be used.

Get Started

Install

• Ubuntu/Debian

$ sudo apt-get install fuse
# python >= 3.7
$ sudo apt-get install python3.8
$ python3.8 -m pip install oxfs

• MacOS
  • Please install osxfuse firstly. links

$ brew install python3
$ mkdir ~/.venv
$ python3 -m venv ~/.venv/oxfs
$ source ~/.venv/oxfs/bin/activate
$ pip install oxfs

Usage

# mount
$ oxfs --host [email protected] --remote-path /home/mark --mount-point mark --cache-path ~/.oxfs --logging /tmp/oxfs.log --daemon --auto-cache

# browse & edit
$ cd mark

# umount
$ umount mark

Help

$ oxfs -h
usage: oxfs [-h] [--host HOST] [--ssh-port SSH_PORT] [--cache-timeout CACHE_TIMEOUT] [--parallel PARALLEL] [--mount-point MOUNT_POINT] [--remote-path REMOTE_PATH]
            [--cache-path CACHE_PATH] [--logging LOGGING] [--daemon] [--auto-cache] [-v]

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           ssh host (example: [email protected])
  --ssh-port SSH_PORT   ssh port (defaut: 22)
  --cache-timeout CACHE_TIMEOUT
                        cache timeout (default: 30s)
  --parallel PARALLEL   parallel (default: equal to cpu count)
  --mount-point MOUNT_POINT
                        mount point
  --remote-path REMOTE_PATH
                        remote path (default: /)
  --cache-path CACHE_PATH
                        cache path
  --logging LOGGING     logging file
  --daemon              daemon
  --auto-cache          auto update cache
  -v, --verbose         debug info

Benchmark

• sshfs: fio -filename=fio20 -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=70 -ioengine=psync -bs=4K -size=10M -numjobs=1 -runtime=100 -group_reporting -name=sshfs
• read: IOPS=149, BW=599KiB/s (613kB/s)(10.0MiB/17093msec)

sshfs: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.29
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=604KiB/s][r=151 IOPS][eta 00m:00s]
sshfs: (groupid=0, jobs=1): err= 0: pid=4611: Thu Mar 17 21:21:21 2022
  read: IOPS=149, BW=599KiB/s (613kB/s)(10.0MiB/17093msec)
    clat (usec): min=127, max=22361, avg=6663.51, stdev=862.88
     lat (usec): min=128, max=22362, avg=6664.35, stdev=862.90
    clat percentiles (usec):
     |  1.00th=[ 5604],  5.00th=[ 5800], 10.00th=[ 5997], 20.00th=[ 6194],
     | 30.00th=[ 6325], 40.00th=[ 6456], 50.00th=[ 6521], 60.00th=[ 6652],
     | 70.00th=[ 6849], 80.00th=[ 7046], 90.00th=[ 7439], 95.00th=[ 7898],
     | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[15533], 99.95th=[20579],
     | 99.99th=[22414]
   bw (  KiB/s): min=  544, max=  624, per=99.65%, avg=597.85, stdev=16.06, samples=33
   iops        : min=  136, max=  156, avg=149.09, stdev= 4.06, samples=33
  lat (usec)   : 250=0.20%
  lat (msec)   : 2=0.04%, 10=99.45%, 20=0.23%, 50=0.08%
  cpu          : usr=0.36%, sys=1.57%, ctx=4977, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=599KiB/s (613kB/s), 599KiB/s-599KiB/s (613kB/s-613kB/s), io=10.0MiB (10.5MB), run=17093-17093msec

• oxfs(first read): fio -filename=fio10 -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=70 -ioengine=psync -bs=4K -size=10M -numjobs=1 -runtime=100 -group_reporting -name=oxfs
• read: IOPS=1036, BW=4147KiB/s (4247kB/s)(10.0MiB/2469msec)

oxfs: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.29
Starting 1 thread
Jobs: 1 (f=1): [r(1)][8.3%][r=517KiB/s][r=129 IOPS][eta 00m:33s]
oxfs: (groupid=0, jobs=1): err= 0: pid=2307: Thu Mar 17 21:25:21 2022
  read: IOPS=1036, BW=4147KiB/s (4247kB/s)(10.0MiB/2469msec)
    clat (usec): min=188, max=259502, avg=960.96, stdev=10153.30
     lat (usec): min=188, max=259502, avg=961.26, stdev=10153.33
    clat percentiles (usec):
     |  1.00th=[   196],  5.00th=[   200], 10.00th=[   202], 20.00th=[   210],
     | 30.00th=[   217], 40.00th=[   219], 50.00th=[   223], 60.00th=[   231],
     | 70.00th=[   239], 80.00th=[   253], 90.00th=[   273], 95.00th=[   297],
     | 99.00th=[   529], 99.50th=[ 53740], 99.90th=[191890], 99.95th=[221250],
     | 99.99th=[258999]
   bw (  KiB/s): min=   15, max= 4798, per=29.71%, avg=1232.75, stdev=2377.03, samples=4
   iops        : min=    3, max= 1199, avg=307.50, stdev=594.38, samples=4
  lat (usec)   : 250=78.55%, 500=20.43%, 750=0.20%
  lat (msec)   : 20=0.04%, 50=0.27%, 100=0.23%, 250=0.23%, 500=0.04%
  cpu          : usr=0.57%, sys=2.84%, ctx=2878, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=4147KiB/s (4247kB/s), 4147KiB/s-4147KiB/s (4247kB/s-4247kB/s), io=10.0MiB (10.5MB), run=2469-2469msec

• oxfs(cached): fio -filename=fio10 -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=70 -ioengine=psync -bs=4K -size=10M -numjobs=1 -runtime=100 -group_reporting -name=oxfs-cached
• read: IOPS=4224, BW=16.5MiB/s (17.3MB/s)(10.0MiB/606msec)

oxfs-cached: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.29
Starting 1 thread

oxfs-cached: (groupid=0, jobs=1): err= 0: pid=3075: Thu Mar 17 21:30:12 2022
  read: IOPS=4224, BW=16.5MiB/s (17.3MB/s)(10.0MiB/606msec)
    clat (usec): min=189, max=1055, avg=233.55, stdev=42.57
     lat (usec): min=189, max=1056, avg=233.81, stdev=42.61
    clat percentiles (usec):
     |  1.00th=[  194],  5.00th=[  200], 10.00th=[  204], 20.00th=[  215],
     | 30.00th=[  217], 40.00th=[  219], 50.00th=[  223], 60.00th=[  227],
     | 70.00th=[  237], 80.00th=[  249], 90.00th=[  269], 95.00th=[  293],
     | 99.00th=[  371], 99.50th=[  429], 99.90th=[  906], 99.95th=[  938],
     | 99.99th=[ 1057]
   bw (  KiB/s): min=17041, max=17041, per=100.00%, avg=17041.00, stdev= 0.00, samples=1
   iops        : min= 4260, max= 4260, avg=4260.00, stdev= 0.00, samples=1
  lat (usec)   : 250=80.82%, 500=18.95%, 750=0.12%, 1000=0.08%
  lat (msec)   : 2=0.04%
  cpu          : usr=1.82%, sys=10.25%, ctx=2671, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=16.5MiB/s (17.3MB/s), 16.5MiB/s-16.5MiB/s (17.3MB/s-17.3MB/s), io=10.0MiB (10.5MB), run=606-606msec

Changelog

• release/0.5.0
  • [Improved] Add cache limit, lru policy.
  • [Removed] Delete ApiServer.
• release/0.4.0
  • [New] Add auto-cache policy to sync file automately.
• release/0.3.2
  • [New] Add user/password auth support.
  • [Improved] Add Config class, use subprocess.Popen.
• release/0.3.1
  • [Fixed] Fix no such file error when write occurred before read.
• release/0.3.0
  • [New] Add daemon support.
  • [New] Upgrade to flask-restx.
  • [Removed] Remove short argument.
  • [Fixed] Fix bugs with git operations.
  • [Fixed] Fix empty file write failed bug.
  • [Fixed] Fix mount permission issue.
• release/0.2.0
  • [New] Add restful API to refresh the cache.
• release/0.1.2
  • [Removed] Remove auto_unmount fuse parameter, some osxfuse do not support it.
  • [Deprecated] Disable the daemon parameter, turn on it in the future.
• release/0.1.1
  • [Added] enable the auto_cache by default.
• release/0.1.0
  • [Fixed] Fix the multi-thread bugs for rename operation.

oxfs's Issues

Add certificate login

The server is for security,
If the server disables password access

oxfs didn't work

$ oxfs -v --host [email protected] --remote-path /home/test/ --mount-point remote_ubuntu/ --cache-path ~/.oxfs --logging oxfs.log --daemon --auto-cache

the directory remote_ubuntu is empty at all, and command runs no error.
oxfs.log

[2022-11-09 11:03:09,156][INFO][transport.py -- _log():1873][Connected (version 2.0, client OpenSSH_8.9p1)]
[2022-11-09 11:03:09,242][INFO][transport.py -- _log():1873][Authentication (publickey) successful!]
[2022-11-09 11:03:09,397][INFO][sftp.py -- _log():158][[chan 0] Opened sftp connection (server version 3)]
[2022-11-09 11:03:09,505][INFO][transport.py -- _log():1873][Connected (version 2.0, client OpenSSH_8.9p1)]
[2022-11-09 11:03:09,593][INFO][transport.py -- _log():1873][Authentication (publickey) successful!]
[2022-11-09 11:03:09,746][INFO][sftp.py -- _log():158][[chan 0] Opened sftp connection (server version 3)]
[2022-11-09 11:04:04,657][INFO][transport.py -- _log():1873][Connected (version 2.0, client OpenSSH_8.9p1)]
[2022-11-09 11:04:04,750][INFO][transport.py -- _log():1873][Authentication (publickey) successful!]
[2022-11-09 11:04:04,927][INFO][sftp.py -- _log():158][[chan 0] Opened sftp connection (server version 3)]
[2022-11-09 11:04:05,034][INFO][transport.py -- _log():1873][Connected (version 2.0, client OpenSSH_8.9p1)]
[2022-11-09 11:04:05,122][INFO][transport.py -- _log():1873][Authentication (publickey) successful!]
[2022-11-09 11:04:05,289][INFO][sftp.py -- _log():158][[chan 0] Opened sftp connection (server version 3)]

ImportError: cannot import name 'cached_property' from 'werkzeug'

~ »  oxfs --help                                                                                                                                                                                                                                                           
Traceback (most recent call last):
  File "/Users/.../.pyenv/versions/3.8.0/bin/oxfs", line 5, in <module>
    from oxfs.oxfs import main
  File "/Users/.../.pyenv/versions/3.8.0/lib/python3.8/site-packages/oxfs/oxfs.py", line 15, in <module>
    from oxfs.apiserver import OxfsApi
  File "/Users/.../.pyenv/versions/3.8.0/lib/python3.8/site-packages/oxfs/apiserver.py", line 6, in <module>
    from flask_restplus import Resource, Api, fields
  File "/Users/../.pyenv/versions/3.8.0/lib/python3.8/site-packages/flask_restplus/__init__.py", line 4, in <module>
    from . import fields, reqparse, apidoc, inputs, cors
  File "/Users/.../.pyenv/versions/3.8.0/lib/python3.8/site-packages/flask_restplus/fields.py", line 17, in <module>
    from werkzeug import cached_property
ImportError: cannot import name 'cached_property' from 'werkzeug' (/Users/.../.pyenv/versions/3.8.0/lib/python3.8/site-packages/werkzeug/__init__.py)

fyi

i reviewed like 40 different remote mounting solutions. Yours was one of the fastest and most stable. Nice wok!

Can't ls a certain folder

Hi. I have an issue that I cannot read the contents of a certain folder. Others work without issues. The only difference in this folder is that it contains 1162 files and it's 19TB although I don't know if it's related with the issue.

[2021-12-27 12:12:45,767][INFO][_internal.py -- _log():225][ * Running on http://127.0.0.1:10010/ (Press CTRL+C to quit)]
[2021-12-27 13:05:24,744][ERROR][fuse.py -- _wrapper():750][Uncaught exception from FUSE operation readdir, returning errno.EINVAL.]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/fuse.py", line 734, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python3.9/dist-packages/fuse.py", line 960, in readdir
for item in self.operations('readdir', self._decode_optional_path(path),
File "/usr/local/lib/python3.9/dist-packages/fuse.py", line 1251, in call
ret = getattr(self, op)(path, *args)
File "/usr/local/lib/python3.9/dist-packages/oxfs/oxfs.py", line 234, in readdir
entries = self.sftp.listdir(path)
File "/usr/local/lib/python3.9/dist-packages/paramiko/sftp_client.py", line 218, in listdir
return [f.filename for f in self.listdir_attr(path)]
File "/usr/local/lib/python3.9/dist-packages/paramiko/sftp_client.py", line 254, in listdir_attr
filename = msg.get_text()
File "/usr/local/lib/python3.9/dist-packages/paramiko/message.py", line 178, in get_text
return u(self.get_string())
File "/usr/local/lib/python3.9/dist-packages/paramiko/py3compat.py", line 161, in u
return s.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 56: invalid continuation byte

Any thoughts about how I can debug this?

cheers

No such file or directory

in Ubuntu 16.04 and Ubuntu 20.04， with python3.8.3，after oxfs --host ...............， mount point fold can't access, ls returned No such file or directory : (mount point folder)，umount or killed oxfs， mount point fold can access normally.

Hello, does version 0.5.3 support account password login? My remote server does not set up the cert login method.

Hello, does version 0.5.3 support account password login? My remote server does not set up the cert login method.

Filesystem full

Hi
First of all congrats for the performance. It's fast!

I'm trying to mount a plex library, which is a 50TB remote filesystem. My problem is that when scanning the lib, the local filesystem (128GB) is filled with oxfs cache and later it's killed by the kernel (out of mem)

Is there a way to limit the cache?

Simple tutorial for Mac User

$ brew install python3
$ mkdir ~/.venv
$ python3 -m venv ~/.venv/oxfs
$ source ~/.venv/oxfs/bin/activate
$ pip install oxfs

readme doc add option description

i think readme doc can add more description of cli option. like this:
-r,--read [path] read [path] file

emacs vim config

setting

vim

set nobackup       "no backup files
set nowritebackup  "only in case you don't want a backup file while editing
set noswapfile     "no swap files

emacs

(setq make-backup-files nil) ; stop creating backup~ files
(setq auto-save-default nil) ; stop creating #autosave# files
(setq create-lockfiles nil)

ImportError: cannot import name '_endpoint_from_view_func' from 'flask.helpers'

error throwed when executing oxfs

ImportError: cannot import name '_endpoint_from_view_func' from 'flask.helpers'

Solved by downgrading flask from 2.0.0 to 1.1.2

ImportError: cannot import name '_endpoint_from_view_func' from 'flask.helpers'

Hello,

Tried to mount my remote dir to my home directory using oxfs but got this:

oxfs --host xsl@siku --remote-path /home/xsl/ --mount-point ./siku --cache-path .oxfs --logging /tmp/oxfs.log --daemon
--auto-cache
Traceback (most recent call last):
File "/Users/xsl/.venv/oxfs/bin/oxfs", line 5, in
from oxfs.oxfs import main
File "/Users/xsl/.venv/oxfs/lib/python3.9/site-packages/oxfs/oxfs.py", line 19, in
from oxfs.apiserver import OxfsApi
File "/Users/xsl/.venv/oxfs/lib/python3.9/site-packages/oxfs/apiserver.py", line 6, in
from flask_restx import Resource, Api, fields
File "/Users/xsl/.venv/oxfs/lib/python3.9/site-packages/flask_restx/init.py", line 5, in
from .api import Api # noqa
File "/Users/xsl/.venv/oxfs/lib/python3.9/site-packages/flask_restx/api.py", line 19, in
from flask.helpers import _endpoint_from_view_func
ImportError: cannot import name '_endpoint_from_view_func' from 'flask.helpers' (/Users/xsl/.venv/oxfs/lib/python3.9/site-packages/flask/helpers.py)

Any idea?

Thanks.

SSH master connection

I tried to connect to a server for which I have a master session running. It seems that offs does not use the SSH configuration that is available. Would it be possible to direct offs to use that connection?