github / vfsforgit

GitHub development on VFSForGit [please fork upstream microsoft/VFSForGit repository instead of this one]

VFS for Git (GitHub partial development fork only)

This repository contains work-in-progress by GitHub's engineering team on the VFS for Git client. All work here will be contributed to the upstream microsoft/VFSForGit repository.

We advise anyone interested in VFS for Git to fork or follow the microsoft/VFSForGit repository instead of this one, as this repository is only a partial clone of the full upstream repo and furthermore may not be regularly updated with the latest changes.

Thank you for your interest in VFS for Git!

Licenses

The VFS for Git source code in this repo is available under the MIT license. See License.md.

vfsforgit's Issues

Wait for FUSE device to be mounted at startup

In our current trial implementation of GVFS.Platform.Linux/LinuxFileSystemVirtualizer, we don't wait after calling the ProjFS.Linux StartVirtualizationInstance() method for our FUSE mount to actually be ready.

This may result in problems where the GVFS provider code proceeds, expecting the mount to be complete, while the asynchronous libfuse/libprojfs threads are still coordinating with the FUSE kernel module to bring up the mount.

We know from experience with the libprojfs test suite that waiting until the mount point's device ID changes is necessary, so we likely should implement the same logic from wait_mount.c in LinuxFileSystemVirtualizer.TryStart():

1. Read the current device ID of the Enlistment.WorkingDirectoryRoot (i.e., the src/ directory) using stat(2).
2. Invoke virtualizationInstance.StartVirtualizationInstance() as is done now.
3. On successful return, poll at short intervals using stat(2) until the device ID changes.
   If the device ID doesn't change after some reasonable timeout period (e.g., 30 seconds?), execute virtualizationInstance.StopVirtualizationInstance() and return an error.

One implementation note involves the complexity of calling stat(2), which is non-trivial; see the current use in MirrorProvider.Linux/LinuxFileSystemVirtualizer and also in GVFS/GVFS.Platform.Linux/LinuxFileSystem. Before we introduce another clone of this code, we should refactor it down to a common set of __xstat64() and __lxstat64() wrappers, perhaps provided by ProjFS.Linux, which can then be used in both LinuxFileSystem and LinuxFileSystemVirtualizer, and elsewhere as needed.

Refactor Mac/Linux versions of FastFetch.NativeUnixMethods with POSIX base class

The FastFetch.NativeUnixMethods class has some of the same flag values and stat(2) calls and struct stat mappings which we've learned need to be refactored as they are different on Linux and BSD/Darwin.

Assuming we may want to run FastFetch on Linux, we should refactor this class along the same lines as GVFS.Platform.POSIX.POSIXFileSystem and its Linux and Mac variants.

Ignore pipe or socket inodes on non-projection events

If the user creates a pipe with mkfifo(1) within the mounted VFSForGit repository, and then tries to rename or remove it, an uninterruptible hang can result which is only resolved by writing data to the pipe on the underlying .gvfs/lower filesystem.

This is due to an attempt to read from the pipe in the VFSForGit GVFS provider event handlers, which looks as thought it ultimately calls the C# File.Exists() method, at a minimum. Either this, or some other code, is trying to perform a read on the pipe, as can be demonstrated by explicitly disabling non-projection event notifications in the ProjFS.Linux layer.

Once #21 lands into the linux-gvfs-provider branch (or further upstream), we can add a stat(2) check at the start of the HandleNonProjEvent() method, just after the check for the provider's pid, and if the path in the event refers to something other than a regular file or directory, we can ignore the event. We can add something like IsReg() and IsDir() to the LinuxNative class in #21.

To ignore the event we should use return perm ? (int)ProjFS.Constants.PROJFS_ALLOW : 0;, and we should also update the return 0 in the check for the provider's pid to do the same. This will ensure that permission events such as the pre-delete event will succeed instead of being denied.

Tracking: Set up unit and functional test CI for Linux

This tracking issue is intended as a roadmap of the steps required to implement the full functional test suite of VFSForGit in Azure DevOps CI for the Linux platform.

• Enable full Linux unit test CI on the features/linuxprototype branch:
  • Enable the PR - Linux - Build and Unit Test pipeline with MirrorProvider build tasks on all PRs to branch (thanks, @jrbriggs!)
  • Merge microsoft#1128, then remove the CPPFLAGS environment variables from the MirrorProvider build tasks.
  • Merge #21 into microsoft#1125.
  • Implement github/libprojfs#71 (#20) (#24) (#27) (#28)
  • Merge microsoft#1125, then enable the VFSForGit build tasks within the PR - Linux - Build and Unit Test pipeline.
  • Implement github/libprojfs#72 (#23)
  • Support case-sensitive file matching (#29) (microsoft#1232) (microsoft#1412) (microsoft#1503) (microsoft#1531)
• Resolve any outstanding problems running the functional test suite on Linux:
  • Enable all relevant functional tests marked Mac-only or Windows-only. (#33)
  • Ensure any necessary PRs are merged into features/linuxprototype (microsoft#1432)
  • Ensure any necessary PRs are merged into master (microsoft#1411) (microsoft#1412) (microsoft#1413) (microsoft#1448) (microsoft#1529) (microsoft#1535)
• Resolve custom Git installation and packaging:
  • Rewrite "git build Linux installer" task to create .deb package within NuGet package and push to MyGet.
  • Add Linux build task to "git - build installers" pipeline.
• Enable full Linux functional test CI on the features/linuxprototype branch:
  • Either an Azure DevOps task group, or, if following the Mac model, a GitHub CI job instead.

Possible additional steps:

• Once functional test CI is successfully working on features/linuxprototype:
  • Refactor out all duplicate code between the Mac and Linux implementations, expanding the POSIX layer (e.g., introducing a ProjFS.POSIX set of classes as per the posix-projfs-virt-instance branch.
  • Merge features/linuxprototype into master and convert both CI jobs to run against master.
  • 🍾 or 🍨, as desired.

Use .vfsforgit instead of .gvfs directories for GVFS provider

In order to avoid conflicts with GNOME VFS (GVfs) and pre-existing .gvfs directories on Linux systems -- not to mention user confusion -- we need the GVFS provider to create and use .vfsforgit* hidden directories instead of its current .gvfs* ones. This could be exclusive to the Linux client, though.

See microsoft#987.

do not lose folders after rename and checkout

The following currently fails on Linux (assuming Foo is an un-hydrated folder present in the repository), because although Bar is removed by the final git checkout, Foo is not restored:

git checkout -b foo
git mv Foo Bar
git add .
git commit -m foo
git checkout master

Determine top-level projection state and pass to libprojfs at mount time

See details in github/libprojfs#71, particularly:

To determine the value of this flag will be an interesting effort, since we do not want to set it on every gvfs mount operation, only the first one. Two options present themselves:

1. Perform a call to StartVirtualizationInstance() at the end of gvfs clone, perhaps in the TryPrepareFolderForCallbacks() method which is run at the very end of the CloneVerb process.
2. At the start of the gvfs mount process, check whether the .gvfs/lower directory contains only .git and .gitattributes, and if so, set the "initial mount" flag to the StartVirtualizationInstance() call.

I am personally inclined toward the second option, as it effectively lifts the "is this directory empty" logic up to the client program's level, where it can determine if the directory is empty according to its own rules (in this case, if it only has the expected post-clone contents), and because it should avoid the need for a full mount/unmount cycle at the end of the cloning step, which feels like a more fragile solution overall.

Tracking: Run VFSForGit functional tests on Linux

In order to run the existing VFSForGit functional test suite, we at least will need a consistent mount process on Linux combined with complete implementation of all lower-level ProjFS.Linux and libprojfs functionality.

Once these components are complete we should be able to run the VFSForGit functional tests and learn what other issues arise from their results.

Known requirements:

• Add all file operation event notifications required by VFSForGit (github/libprojfs#63)
  • microsoft#1070
• Wait for FUSE device to be mounted at startup (#12)
  • #21
• Implement DeleteFile and UpdatePlaceholderIfNeeded in ProjFS.Linux (#13)
  • microsoft#1128
• Fix event handling bugs in ProjFS.Linux (microsoft#1189)
• Determine top-level projection state and pass to libprojfs at mount time (#20) (#24)
  • Set initial projection state on backing store at first mount (github/libprojfs#71) (github/libprojfs#73)
  • Determine initial projection state in provider rather than ProjFS layer (#27) (#28)
• Ensure GVFS provider only accesses .git on lower storage filesystem (#15)
  • microsoft#1074
  • microsoft#1104
  • microsoft#1119
  • microsoft#1123
• Build on Linux (microsoft#1125)
• Resolve permissions error on git add (#23)
  • Detect and set projection state on read-only files (github/libprojfs#72)
• Support case-sensitive file matching (#29)
  • microsoft#1232
  • microsoft#1412
  • microsoft#1503
  • microsoft#1531
• Enable all relevant functional tests marked Mac-only or Windows-only (#33)
• Build GVFS.FunctionalTests and run all functional tests on Linux (#18)
  • microsoft#1411
  • microsoft#1413
  • microsoft#1432
  • microsoft#1448
  • microsoft#1529
  • microsoft#1535

Determine initial projection state in provider rather than ProjFS layer

The IsUninitializedMount() method introduced in 0a4b8a1 works well when used with the GVFS provider, where we expect the working directory "backing store" (i.e., the lower projfs directory) to contain a .git directory and a .gitattributes file. However, when used with MirrorProvider, the result is that no projection of the top-level directory occurs, because these Git artifacts do not exist, so IsUninitializedMount() always returns false.

Moving the "is initial mount?" logic to the provider level would allow for each provider to determine the question appropriately; in the case of MirrorProvider, an empty top-level directory should indicate an initial mount.

This will likely require adding a bool "initial mount" argument to the StartVirtualizationInstance() method signature, as suggested in github/libprojfs#71.

Implement DeleteFile and UpdatePlaceholderIfNeeded in ProjFS.Linux

The DeleteFile(), UpdatePlaceholderIfNeeded(), and ReplacePlaceholderFileWithSymLink() methods are currently stubbed out.

They are used by the GVFS provider when it needs to add/remove files to ensure the working directory matches a newly checkout-out Git branch, because Git (when used with VFSForGit) does not do so itself -- Git only manages files which have been directly modified by the user; all others have to be updated by VFSForGit.

The Mac implementation of these methods ultimately uses normal C/C++ syscalls like unlink()/rmdir() to do this, but has a number of TODO comments suggesting that races with file hydration are yet to be handled properly.

We could potentially implement these methods using additional libprojfs functions, e.g., projfs_delete_file(), etc., which would acquire a lock using the internal get_path_userdata()/finalize_userdata() functions on the file/directory or parent directory, as necessary, to avoid races with the hydration performed by projfs_fuse_proj_locked().

However, we should avoid excess locking where possible and rely on the Linux VFS to handle most contention issues, and it's also valid for any program (Git or otherwise) to attempt to add/update/remove files under our mount point just using regular POSIX syscalls. So an initial implementation should likely follow the Mac model and simply use POSIX calls like unlink(), etc., without adding any functions to the libprojfs API.

remove all sources of uppercase SHA1 values

I would be ideal to eliminate the need to down-case SHA1 values in
GitObjects and GitRepo, which could be achieved if we can ensure all sources of such SHA strings are always lowercase.

At least one such source is presumably in Sha1Id, but these may be persisted (e.g., in databases or files) and may also arrive from other sources.

If eliminating all sources of uppercase SHAs is difficult or impossible, then be current down-case-on-demand approach is sufficient, but an investigation would be desirable before concluding that permanently.

Search PATH for VFSForGit-enabled Git binary

Instead of expecting our VFSForGit-enabled git binary to be installed in /usr/local/bin, as per the macOS implementation, we should look through the locations in our PATH environment variable and check for a git whose version matches the one we need, or fail if the first found one does not.

We should be able to use Environment.GetEnvironmentVariable() to acquire the PATH variable, at least.

resolve problem with "git add" on mounted repos

When trying to run git add on a changed file within a VFSForGit repository mounted using the latest linux-gvfs-provider branch, we receive the following errors:

error: unable to write sha1 filename: .git/objects/../...: Permission denied
error: ... failed to insert into database
error: unable to index file ...
fatal: updating files failed

Triggered by github/libprojfs#72.

Support case-sensitive file matching on Linux

As noted in microsoft#1125, we will need to refactor a number of points in the existing GVFS provider code to support case-sensitive file names on Linux.

Implement GVFS service registry

The gvfs mount command wants to register each new mount with a system service daemon, presumably so that the mounts can be auto-mounted upon restarted. We will need to consider how best to implement such a GVFS.Service daemon for Linux.

Note that in the interim, we are using the GVFS_UNATTENDED environment variable to disable the gvfs command's attempts to contact the GVFS.Service process.

Ensure GVFS provider only accesses .git on lower storage filesystem

At present, mounting the GVFS provider stalls for a potentially long period of time because the initial enumeration callback for the top-level directory of the mount point (src/) iterates through the list of entries in that directory, and for each one, tries to read the "blob state" by looking in src/.git (which is the path currently returned by this.enlistment.LocalObjectsRoot). Each of these requests goes to libprojfs, which tries to enumerate the parent directory, which (as intended) times out waiting for the per-inode hydration lock. Eventually the GVFS provider proceeds, and correctly populates the top-level directory, but only after a potentially long series of timeouts.

To address this we propose to find a way to, on Linux only, alter the GVFS enlistment configuration such that all paths to the .git directory resolve to .gvfs/lower/.git instead of src/.git. Further, we should at the same time determine how best to set the "storage root" path of .gvfs/lower in a common location (i.e., somewhere in an Enlistment class), even though this is a platform-specific path and concept which will have no meaning on the Windows and Mac platforms.

Change .gvfsCache folder to .vfsforgitCache

The GVFS provider, by default, creates a ~/.gvfsCache folder during cloning, or a different location may be specified with --local-cache-path. As with #9, we should change the default name of this folder to .vfsforgitCache, at least on Linux.

See also microsoft#987 and microsoft#72.

Handle unmount failures gracefully

After running the functional test suite for a while, and encountering a hard-stop error, a number of "stale" mounts are left behind:

/dev/fuse on /GVFS.FT/test/238704f51ee541ec9049/src type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
/dev/fuse on /GVFS.FT/test/bd7025c836fa4cbe94d7/src type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
/dev/fuse on /GVFS.FT/test/f0fb2de381ea4a81a9a1/src type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

These may result when a functional test fails and doesn't clean up properly, or tries to clean up and unmount but a running process holds an open file handle within the mount.

The latter issue is one we will need to consider -- what should the action of gvfs unmount be when we cannot actually unmount the FUSE device?

enable on Linux all relevant functional tests marked Mac/Windows-only

Some functional tests currently marked with either MacOnly or WindowsOnly categories will also be relevant on Linux, e.g., GitCommandsTests.UpdateFileModeOnly().

We need to audit all of the MacOnly and WindowsOnly tests, as well as the MacTODO tests, and ensure we enable all of those which are relevant for the Linux platform and implementation.

Prevent deadlocks when GVFS provider triggers callback events

If the GVFS provider process itself changes any files in the mounted src/ directory, this will trigger a callback notification event from libprojfs to the provider, which could result in a deadlock (apparently).

The ProjFS.Mac code avoids this by ignoring file operations generated by the provider, and implements this filter at the kext level.

It would be preferable (from a libprojfs API design standpoint) if we can filter these events at the ProjFS.Linux VirtualizationInstance level -- that is, we send a callback to the C# but it immediately returns without triggering any higher-level code in the provider logic itself. That should be enough to prevent deadlocks (we hope!) while keeping the API simple; this way the ProjFS.Linux C# code can just filter for its own process's pid in Handle[Non]ProjEvent() handlers.