github / libprojfs

Linux projected filesystem library

libprojfs

A Linux projected filesystem library, similar in concept to the Windows Projected File System and developed in conjunction with the VFSForGit project.

While the libprojfs C library may also be used independently of VFSForGit, our primary goal is to enable users to run the VFSForGit client on a Linux system.

PLEASE NOTE: At present libprojfs is undergoing rapid development and its design and APIs may change significantly; some features are also incomplete, and so we do not recommend libprojfs for any production environments.

However, we wanted to share our progress to date, and we hope others are as excited as we are at the prospect of running the VFSForGit client on Linux in the not-too-distant future!

Current Status

Branch	Functional Tests
master

The library is under active development and supports basic directory projection, with additional features added regularly.

We will only make a first, official versioned release once initial development is complete.

A libprojfs filesystem can currently be used to mount a VFSForGit MirrorProvider test client; we expect to continue our pre-release development until the VFSForGit client proper is also functional with libprojfs, at which point we may make an initial tagged release.

Design

The libprojfs library is designed to function as a stackable Linux filesystem supporting a "provider" process which implements custom callbacks to populate files and directories on demand. This is illustrated in the context of a VFSForGit provider below:

Actual file storage is delegated to a "lower" storage filesystem, which may be any Linux filesystem, e.g., ext4. At the time when a libprojfs filesystem mount is created, there may be no files or directories in the lower filesystem yet.

As normal filesystem requests are made within the libprojfs mount (e.g., by ls or cat), libprojfs intercepts the requests and queries the provider process for the actual contents of the file or directory. The provider's response is then written to the lower filesystem so future accesses may be satisfied without querying the provider again.

See our design document for more details.

We anticipate that whether libprojfs remains a FUSE-based library, or becomes a libfuse-like interface to a Linux kernel module, it may be useful for purposes other than running a VFSForGit client.

For this reason, we have tried to ensure that our native event notification API is aligned closely with the Linux kernel's fanotify/inotify/fsnotify APIs.

Getting Started

Please see our detailed Build and Installation page for step-by-step instructions on how to build and install libprojfs and its dependencies on Linux, and (optionally) also build and run the VFSForGit MirrorProvider test utility with libprojfs.

So long as libprojfs remains based on FUSE, the primary dependency for libprojfs is the user-space libfuse library, as well as having the Linux fuse kernel module installed.

Support for user.* extended attributes must also be enabled on the "lower" storage filesystem. Your Linux kernel should have been compiled with the necessary configuration options (e.g., CONFIG_EXT2_FS_XATTR for ext2/3/4) and the underlying filesystem you use for libprojfs should be mounted with appropriate settings (e.g., user_xattr for ext2/3/4).

If you are using a Docker container, you may need to ensure the fuse kernel module is installed on your host OS, and user.* extended attributes are supported on your host's filesystem. See the Using Docker containers section for more details.

To test libprojfs with the Microsoft VFSForGit MirrorProvider (as we do not support the primary VFSForGit GVFS provider yet), .NET Core must be installed and parts of the VFSForGit project built. See the Building and Running MirrorProvider section for details.

Contributing

Thank you for your interest in libprojfs!

We welcome contributions; please see our CONTRIBUTING guidelines and our contributor CODE_OF_CONDUCT.

Licensing

The libprojfs library is licensed under the LGPL v2.1.

See the NOTICE file for a list of other licenses used in the project, and the comments in each file for the licenses applicable to them.

Development Roadmap

We are developing the libprojfs library first, using FUSE to prototype and test its performance, before migrating functionality into a Linux kernel module (assuming that proves to be necessary to meet our performance criteria).

For more details on the planned development phases, see our design document.

Authors

The libprojfs library is currently maintained and developed by several members of GitHub's Engineering organization, including:

• @chrisd8088
• @kivikakk
• @wrighty

You can also contact the GitHub project team at [email protected].

libprojfs's Issues

Add all file operation event notifications required by VFSForGit

Specifically, we likely need to add event notifications for the FileModifed and HardLinkCreated operations; see the use of these in, e.g., the macOS and generic handlers registered by the GVFS provider.

We should likely map these to Linux equivalents as PROJFS_MODIFY and PROJFS_CREATE_SELF | PROJFS_ONLINK, where the ONLINK flag is used like the ONDIR one:

#define PROJFS_MODIFY 0x00000002

#define PROJFS_ONLINK himask(0x1000)

We're trying to align with fanotify.h in all this, but it's not perfect (yet!), and we're also tracking Amir Goldstein's superblock root watch work.

Detect and set projection state on read-only files

The github/VFSForGit#23 issue stems from an internal issue whereby the git add command first creates a temporary file (e.g., .git/objects/19/tmp_obj_wdSBSa) with read-only file permissions, and then requests a hard link be created to another path (e.g., .git/objects/19/0a18037c64c43e6b11489df4bf0b9eb6d2c9bf).

The projfs_op_link() operation begins by making sure all the parent directories and the source file are fully projected, and in particular, calls project_file() on the source path. This in turn calls acquire_proj_state_lock() with the O_RDWR flag because we are hoping to be notified if the path actually corresponds to a directory; in such a case, the openat(2) that function would return EISDIR.

Because the source file is read-only, however, the openat(2) fails with EACCES, upon receipt of which the op_link operation pre-emptively returns with an error.

Simulating this behaviour is straightforward on a test mount; it occurs not only on hard link operations but move/rename as well:

% touch FOO
% chmod 444 FOO
% ln FOO BAR
ln: failed to create hard link 'BAR' => 'FOO': Permission denied
% mv FOO BAR
mv: cannot move 'FOO' to 'BAR': Permission denied

Furthermore, our call to fsetxattr(2) will fail on read-only files, which inhibits our ability to set (and remove) our projection state flag.

We likely need to implement acquire_proj_state_lock() by only opening files with O_RDONLY, acquiring our flock(2), then testing with fstat(2) to determine if the file is not a regular file, and also setting/resetting file permissions, if necessary, around our fsetxattr(2) call.

---

As a side note, along with the challenges we've encountered in performing locking around projection state transitions (which would ideally be handled within our inode structures instead of with flock()), this may be another case where having low-level FUSE API access, or kernel-level access, would be valuable.

Check for xattr support at mount time

Also check for sparse file support (if needed for #6).

Event notifications on file/dir rename operations

Docker setup failed

when run ./projfs setup:

Step 4/4 : RUN chmod 0777 /data
 ---> Using cache
 ---> 52384ab289d6
Successfully built 52384ab289d6
Successfully tagged github/vfs-linux:latest
>> docker run -u 1001 --pid=host --rm --name projfs-vfs -v /data00/home/tanhaiyang/libprojfs/docker/VFSForGit:/data/vfs/src -v /data00/home/tanhaiyang/libprojfs/docker/build/packages:/data/vfs/packages -v /data00/home/tanhaiyang/libprojfs/docker/build/BuildOutput:/data/vfs/BuildOutput github/vfs-linux env DOTNET_CLI_TELEMETRY_OPTOUT=1 DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 NUGET_XMLDOC_MODE=skip dotnet restore MirrorProvider/MirrorProvider.sln /p:Configuration=Debug.Linux --packages ../packages
MSBUILD : error MSB1009: Project file does not exist.
Switch: MirrorProvider/MirrorProvider.sln
./projfs:95:in `system': failed to run 'docker run -u 1001 --pid=host --rm --name projfs-vfs -v /data00/home/tanhaiyang/libprojfs/docker/VFSForGit:/data/vfs/src -v /data00/home/tanhaiyang/libprojfs/docker/build/packages:/data/vfs/packages -v /data00/home/tanhaiyang/libprojfs/docker/build/BuildOutput:/data/vfs/BuildOutput github/vfs-linux env DOTNET_CLI_TELEMETRY_OPTOUT=1 DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 NUGET_XMLDOC_MODE=skip dotnet restore MirrorProvider/MirrorProvider.sln /p:Configuration=Debug.Linux --packages ../packages (RuntimeError)
	from /data00/home/tanhaiyang/libprojfs/docker/project.rb:71:in `run'
	from /data00/home/tanhaiyang/libprojfs/docker/project.rb:50:in `block in command'
	from /data00/home/tanhaiyang/libprojfs/docker/project.rb:49:in `each'
	from /data00/home/tanhaiyang/libprojfs/docker/project.rb:49:in `command'
	from ./projfs:206:in `<main>'
tanhaiyang@n227-087-045:~/libprojfs/docker$ docker run -u 1001 --pid=host --rm --name projfs-vfs -v /data00/home/tanhaiyang/libprojfs/docker/VFSForGit:/data/vfs/src -v /data00/home/tanhaiyang/libprojfs/docker/build/packages:/data/vfs/packages -v /data00/home/tanhaiyang/libprojfs/docker/build/BuildOutput:/data/vfs/BuildOutput github/vfs-linux env DOTNET_CLI_TELEMETRY_OPTOUT=1 DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 NUGET_XMLDOC_MODE=skip dotnet restore MirrorProvider/MirrorProvider.sln /p:Configuration=Debug.Linux --packages ../packages
MSBUILD : error MSB1009: Project file does not exist.
Switch: MirrorProvider/MirrorProvider.sln 

Test use of flock(2)

We need to ensure that our flock(2)-based synchronisation actually works. We flock(2) in two scenarios:

• when hydrating a directory with placeholders; and,
• when hydrating a placeholder file.

If our synchronisation failed, we would expect that attempting to enumerate an unhydrated directory or access an unhydrated placeholder would block while the provider does the work, but that a second concurrent attempt would not block and wait for the first, and instead would overlap with it. This could cause data corruption, or a crash.

To test, we create a provider which, on projection request, opens a file with O_CREAT|O_EXCL, sleeps 1 second, then removes the file. We then try to list the directory contents twice, concurrently. If the file open fails with EEXIST (in one of the attempts), this implies the provider was called to project the same directory twice concurrently, and that libprojfs didn't synchronise access correctly.

replace flock with internal per-inode lock

Our current implementation uses flock(2) to attempt to acquire an (advisory) exclusive lock on an inode while determining its projection state; however, as noted, this may interfere with clients' independent use of flock(2).

The VFSForGit application and .NET Core are an example of such a client; in particular, the UpdatePlaceholderTests functional tests are currently failing because they acquire file handles using the FileShare.Delete mode, which uses a flock() mode of LOCK_SH. The GitIndexProjection.UpdateOrDeleteFilePlaceholder() method then invokes the DeleteFile() method of the Linux VirtualizationInstance, which attempts a simple File.Delete() but receives EAGAIN from libprojfs's failed flock(2) call using LOCK_EX | LOCK_NB.

Replacing the use of flock(2) with an internal inode-to-lock mapping in libprojfs would alleviate this type of contention for locks with all clients, including VFSForGit.

Note that lock contention of this type may also be the reason we have to handle EAGAIN return codes from attempts to write to files in other places in the VFSForGit functional test suite; see, for example, (github/VFSForGit@88cc76a).

It may also be the cause of the apparently transient failures sometimes encountered with the GitCommands.StatusTests.CreateFileWithoutClose() and GitCommands.StatusTests.WriteWithoutClose() tests, which have been seen to report Resource temporarily unavailable (i.e., an EAGAIN error code) while attempting to clean up their test files. For example:

Failed : GVFS.FunctionalTests.Tests.GitCommands.StatusTests(False).CreateFileWithoutClose()
clean -d -f -x Errors Lines
clean -d -f -x Errors Lines counts do not match. was: 1 expected: 0
Extra: warning: failed to remove CreateFileWithoutClose.md: Resource temporarily unavailable
   at GVFS.Tests.Should.EnumerableShouldExtensions.ShouldMatchInOrder[T](IEnumerable`1 group, IEnumerable`1 expectedValues, Func`3 equals, String message) in GVFS/GVFS.Tests/Should/EnumerableShouldExtensions.cs:line 119
   at GVFS.FunctionalTests.Tools.GitHelpers.ErrorsShouldMatch(String command, ProcessResult expectedResult, ProcessResult actualResult) in GVFS/GVFS.FunctionalTests/Tools/GitHelpers.cs:line 190
   at GVFS.FunctionalTests.Tests.GitCommands.GitRepoTests.RunGitCommand(String command, Boolean ignoreErrors, Boolean checkStatus) in GVFS/GVFS.FunctionalTests/Tests/GitCommands/GitRepoTests.cs:line 194
   at GVFS.FunctionalTests.Tests.GitCommands.GitRepoTests.TestValidationAndCleanup(Boolean ignoreCase) in GVFS/GVFS.FunctionalTests/Tests/GitCommands/GitRepoTests.cs:line 116
   at GVFS.FunctionalTests.Tests.GitCommands.GitRepoTests.TearDownForTest() in GVFS/GVFS.FunctionalTests/Tests/GitCommands/GitRepoTests.cs:line 100

Failed : GVFS.FunctionalTests.Tests.GitCommands.StatusTests(False).WriteWithoutClose()
reset --hard -q HEAD Errors Lines
reset --hard -q HEAD Errors Lines counts do not match. was: 2 expected: 0
Extra: error: unable to unlink old 'Readme.md': Resource temporarily unavailable
Extra: fatal: Could not reset index file to revision 'HEAD'.
   at GVFS.Tests.Should.EnumerableShouldExtensions.ShouldMatchInOrder[T](IEnumerable`1 group, IEnumerable`1 expectedValues, Func`3 equals, String message) in GVFS/GVFS.Tests/Should/EnumerableShouldExtensions.cs:line 119
   at GVFS.FunctionalTests.Tools.GitHelpers.ErrorsShouldMatch(String command, ProcessResult expectedResult, ProcessResult actualResult) in GVFS/GVFS.FunctionalTests/Tools/GitHelpers.cs:line 190
   at GVFS.FunctionalTests.Tests.GitCommands.GitRepoTests.RunGitCommand(String command, Boolean ignoreErrors, Boolean checkStatus) in GVFS/GVFS.FunctionalTests/Tests/GitCommands/GitRepoTests.cs:line 194
   at GVFS.FunctionalTests.Tests.GitCommands.GitRepoTests.TestValidationAndCleanup(Boolean ignoreCase) in GVFS/GVFS.FunctionalTests/Tests/GitCommands/GitRepoTests.cs:line 115
   at GVFS.FunctionalTests.Tests.GitCommands.GitRepoTests.TearDownForTest() in GVFS/GVFS.FunctionalTests/Tests/GitCommands/GitRepoTests.cs:line 100

expand mirror test suite to cover all file/dir/fs ops

projection/hydration of files on open() and related file ops

Add projection of files to the directory projection work in #5; dependent on test suite work in #7.

Avoid FUSE fork using xattrs and flock(2)

Project status update?

@chrisd8088 @kivikakk @wrighty

Hi folks, could you please provide a quick update on this project's status?

It would be nice to have a GVFS solution for linux and especially containers.

Cheers.

Implement pre-modification state flag for hydrated files

Per our design, we need a third state for files which have been hydrated with their contents, but not modified by the user. We can implement this by setting our user.projection.empty flag to n, and then only removing the xattr if the file is opened for writing.

This will allow us to detect and send events at the "convert to full (i.e., modified)" state transition (when the file is opened for writing), akin to how the VFSForGit Mac kext sends PreConvertToFull events when it receives a KAUTH_VNODE_WRITE_DATA authorization request on a file which has been hydrated but still has a org.vfsforgit.xattr.file xattr. After sending this event, the Mac user-space library then removes the xattr.

We don't need a second xattr, because unlike the VFSForGit kext, which uses a spare BSD file flag to mark un-hydrated empty files, we're already using an xattr for that purpose, and can simply give it a different value (n) upon hydration, and then only remove it if the file is ever opened for writing.

Store GVFS metadata in xattrs on placeholder/empty files

Enhance the C# caller code (ProjFS.Linux) and the libprojfs API as necessary to support storing the GVFS attributes contentId and providerId as extended attributes on empty/placeholder projected files.

support building against recent libfuse v2.x or older v3.x

Requiring use of libfuse version 3.x may be barrier to adoption for some for whom their Linux (or BSD?) distribution only has version 2.x installed by default. This may be particularly true for those running Long-Term Support distro versions.

What would be great (not urgent, but a nice-to-have) would be to adapt our code to the best available FUSE library, either auto-detected by autoconf or overridden with a --with-fuse=/path/to/lib sort of configure flag.

spoof events on libfuse hidden files, or use top-level hidden directory

Because we do not, at present, enable the hard_remove libfuse option, when a file is renamed or deleted while a process has an open file descriptor for it, the file is actually just renamed as .fuse_hidden... until the handle is removed.

This has two consequences -- one is that we deliver PROJFS_MOVE* and PROJFS_DELETE* events on the hidden filename, and the other is that the hidden file then prevents further actions, such as deletion of the parent directory.

We should consider spoofing events such that we do not deliver them for .fuse_hidden... files, and instead report PROJFS_DELETE* events from the rename() file op when the file is being renamed to a hidden filename, while noting this is insufficient to allow further operations such as deletion of the parent directory -- which may manifest in directories "left over" after a git checkout while using VFSForGit.

An alternative would be to investigate how to make libfuse instead place the hidden file in a directory at the top of the filesystem, e.g., in <mount point>/.fuse_hidden/. We could also deliberately hide that directory from the clients, if desired.

projection/hydration of directories on opendir() and related dir ops

Migrate VFS API support into VFSForGit repo

There's a WIP up for discussion at github/VFSForGit#3.

Refactor test suite to support projection

This should be a PR which precedes merging #5 and #6 as their tests will depend on these changes.

Add Azure Pipelines CI badge to README

Set initial projection state on backing store at first mount

To utilize libprojfs with VFSForGit, we need client-managed mechanism by which to set the initial projection state on the top-level mount point directory. Currently, we do this in libprojfs by testing, at mount time, whether the mount point directory is empty.

However, when used with VFSForGit, we want to set the flag once sometime after the user runs gvfs clone, at which point the directory will already have .git and .gitattributes in it, so our default check_dir_empty() test will not work.

A simple fix is to have the VFSForGit C# code "manually" set the user.projection.empty flag to y on the directory after gvfs clone runs. But this breaks our API abstraction, which assumes this flag is managed by libprojfs.

From the libprojfs perspective, an option which aligns well with libfuse is to accept an args array of option arguments, which are parsed at the time the FUSE filesystem handle is created. Similarly, our projfs_new() function could accept a list of arguments to parse, including (potentially) supported FUSE arguments like -o debug which could be passed down to fuse_new() instead of our current fixed compile-time argument array to FUSE.

If we implement such an option (e.g., -o init or --initial-mount or something like that) following, for instance, the example of the libfuse mount utility, then we will want to pass an additional bool "initial mount" flag argument to StartVirtualizationInstance() which can be converted into the appropriate argument array and passed down to projfs_new().

To determine the value of this flag will be an interesting effort, since we do not want to set it on every gvfs mount operation, only the first one. Two options present themselves:

1. Perform a call to StartVirtualizationInstance() at the end of gvfs clone, perhaps in the TryPrepareFolderForCallbacks() method which is run at the very end of the CloneVerb process.
2. At the start of the gvfs mount process, check whether the .gvfs/lower directory contains only .git and .gitattributes, and if so, set the "initial mount" flag to the StartVirtualizationInstance() call.

I am personally inclined toward the second option, as it effectively lifts the "is this directory empty" logic up to the client program's level, where it can determine if the directory is empty according to its own rules (in this case, if it only has the expected post-clone contents), and because it should avoid the need for a full mount/unmount cycle at the end of the cloning step, which feels like a more fragile solution overall.

See also github/VFSForGit#20.

Use `.vfsforgit*` instead of `.gvfs*` directories for GVFS provider.

In order to avoid conflicts with GNOME VFS (GVfs) and pre-existing .gvfs directories on Linux systems -- not to mention user confusion -- we need the GVFS provider to create and use .vfsforgit* hidden directories instead of its current .gvfs* ones. This could be exclusive to the Linux client, though.