presslabs / gitfs

Version controlled file system

gitfs

Welcome to GitFS

gitfs is a FUSE file system that fully integrates with git. You can mount a remote repository's branch locally, and any subsequent changes made to the files will be automatically committed to the remote.

gitfs was developed by the awesome engineering team at Presslabs, a Managed WordPress Hosting provider.

What's its purpose?

gitfs was designed to bring the full powers of git to everyone, no matter how little they know about versioning. A user can mount any repository and all their changes will be automatically converted into commits. gitfs will also expose the history of the branch you're currently working on by simulating snapshots of every commit.

gitfs is useful in places where you want to keep track of all your files, but at the same time you don't have the possibility of organizing everything into commits yourself. A FUSE filesystem for git repositories, with local cache.

Installing

We provide packages for the major Ubuntu releases and MacOS, but you can find community packages for most of popular Linux distributions. If you want to build packages for a distribution, or you already did, please contact us and we'll list it here.

Ubuntu 18.04+

sudo add-apt-repository ppa:presslabs/gitfs
sudo apt-get update
sudo apt-get install gitfs

MacOS

brew install gitfs

Pip

We also publish a package to PyPI, which can be installed via pip using the following commmand:

pip install gitfs

Usage

You can mount a remote or local repository easly, just by providing the repository to clone and a directory used to mount.

gitfs http://your.com/repository.git /mount/directory

The entire filesystem can be tweaked when mounting it, using a set of options.

gitfs [email protected]:user/repo.git /mypath -o
repo_path=/tmp/path,branch=dev,log=-,debug=true,foreground=true,fetch_timeout=0.1,merge_timeout=0.1...

For an entire list of options, you can check the arguments page.

Features

• Automatically commits changes: create, delete, update files and their metadata
• Browse through working index and commit history
• Merges with upstream by automatically accepting local changes
• Caching commits reduces the memory footprint and speeds up navigation
• Reduces the number of pushes by batching commits

Development

You can find more documentation on gitfs homepage.

Contributing

Development of gitfs happens at http://github.com/presslabs/gitfs.

Issues are tracked at http://github.com/presslabs/gitfs/issues.

The Python package can be found at https://pypi.python.org/pypi/gitfs/.

You are highly encouraged to contribute with code, tests, documentation or just sharing experience.

Please see CONTRIBUTING.md.

License

This project is licensed under Apache 2.0 license. Read the LICENSE file in the top distribution directory for the full license text.

gitfs's Issues

Browse working tree

• clone in a directory specified at mount point
• follow branch specified at mountpoint

Strange race condition on commit/push

for i in {1..100};
do echo "some test" >> current/me; touch current/tmp$i; mv current/tmp$i current/tmp_$i;
done

Will raise:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/wok/presslabs/gfs/gitfs/worker/base.py", line 20, in run
    self.execute(jobs)
  File "/home/wok/presslabs/gfs/gitfs/worker/commit.py", line 25, in execute
    message = self._index_paths(add, 'add', "Updated %s items")
  File "/home/wok/presslabs/gfs/gitfs/worker/commit.py", line 38, in _index_paths
    getattr(self.repository.index, operation)(path)
KeyError: 'tmp1'

On open and access, raise specific error

When we try to read a file from a ReadView, in open we should check for write flags(os.O_WRONLY | os.O_RDWR | os.O_APPEND | os.O_TRUNC | os.O_CREAT), otherwise, raise EROFS and in access, check for right mode(R_OK|F_OK), otherwise, raise EACCES

Restructure code

• a lot of decorators.py
• remove duplicate code
• move repository.py from utils
• restructure accept_mine.py

Folder creation create a `.keep` file

Git works only with files. In order to add a folder to VCS a placeholder file should be created. This file should not be deletebale. Only deleting the folder deletes this file.

Push on write

Allow mounting repos that do not end with .git

Update the regex from _get_repo (Repository).

Read files at certain commits

Read files from the /history file using something like git cat-file blob HEAD:README.md

Writes on `/history` folder should return `Read only filesystem`

All timeout should be configurable from cli

Implement merging

Restructure tests

Create file

Enable file writes. Commits should be done when the file handle is closed.

Refactor Makefile

Implement unlink

getattr fails with multiple directories

Put the filesystem in read-only mode if pull/push fails

If the remote is down we should put ourselves in read-only mode until we are recovering.

Commit/push on fsync

refactor git object searching

Filter /history paths so that they are valid.

E.g.: from a directory corresponding to a CommitView right now you can change dir to an non-existent directory and it would work. That behaviour should not be allowed.

st_mtime for HistoryView is not UTC

Add tests for PassthroughFuse

Browse directory structure

ls -l should display the file mode for working directory as well as /history

• CurrentView
• HistoryView
• IndexView
• CommitView

Bad address for commit and history view

Cache the commit tree in memory

The commit tree should be kept in memory since we always know when that is modified.

Refactor paths from repo

• give a name to repo
• fix self.repo_path

Improve cache system

Cache the entire commit tree

Allow permission modifications

Implement retry for failing jobs

Implement fsync and fsyncdir

Owner modifications should return ENOSYS

Duplicate code in Repository._get_git_object and Repository._get_git_object_type

When we mount it as root, other users don't have permissions to see the mount point

Tried:

http://unix.stackexchange.com/questions/37168/unable-to-use-o-allow-other-with-sshfs-option-enabled-in-fuse-conf

http://www.elfnet.org/2010/06/29/sshfs-failed-to-open-devfuse-permission-denied/

Batch pushes

• Do git push if writes are idling more that coalesce_writes seconds or after coalesce_timeout is hit

Add the class diagram from #58 to the official docs.

Git pull should be done in background

• Do git pull on readdir if the cache is older than dir_cache seconds
• Do git pull and cache every cache seconds

Writes on file time should return ENOSYS

.git directory still visible in current

Specify the owner/group at mount

All files should be owned by a user/group that is specified at mount. Git doesn't have the notion of owner.

Merge should be done periodically

Don't commit on empty index

Make use of user/group specified at mount point

Read file from the current working directory

Make mount entry point

python -m gitfs.mount should be the entry point for mounting a folder.

We need a new thread for commit/push

Finish the tests for the Repository class

Methods that still need to be tested:

• commit
• get_blob_data
• get_blob_size
• get_git_object
• get_git_object_type

Refactor View and PasstroughFuse

View -> inherit from Operations
PassthroughFuse -> inherit from View
Current -> inherit from PassthroughFuse

Return write failure after certain file size

Browse commits

Commits should be available under /history root folder. They should be organised like this:

gitium-fs
├── current
└── history
    ├── 2014-06-30
    │   └── 14-30-40-33f9603
    │       └── INFO
    └── 2014-07-03
        ├── 14-30-40-6f04e04
        │   └── INFO
        └── 14-30-40-a3f09de
            └── INFO

They are grouped by commit day (UTC time) and each commit short sha1 is prefiex with the commit time (UTC).

Each commit folder has an INFO file containing the full commit description (eg. from git log).

When you stop the script or unmount, the repo should be deleted also