twpayne / go-vfs

Package vfs provides an abstraction of the os and io packages that is easy to test.

go-vfs

Package vfs provides an abstraction of the os and io packages that is easy to test.

Key features

• File system abstraction layer for commonly-used os and io functions from the standard library.

• Powerful and easy-to-use declarative testing framework, vfst. You declare the desired state of the filesystem after your code has run, and vfst tests that the filesystem matches that state. For a quick tour of vfst's features, see the examples in the documentation.

• Compatibility with github.com/spf13/afero and github.com/src-d/go-billy.

Quick start

vfs provides implementations of the FS interface:

// An FS is an abstraction over commonly-used functions in the os and io
// packages.
type FS interface {
    Chmod(name string, mode fs.FileMode) error
    Chown(name string, uid, git int) error
    Chtimes(name string, atime, mtime time.Time) error
    Create(name string) (*os.File, error)
    Glob(pattern string) ([]string, error)
    Lchown(name string, uid, git int) error
    Link(oldname, newname string) error
    Lstat(name string) (fs.FileInfo, error)
    Mkdir(name string, perm fs.FileMode) error
    Open(name string) (fs.File, error)
    OpenFile(name string, flag int, perm fs.FileMode) (*os.File, error)
    PathSeparator() rune
    RawPath(name string) (string, error)
    ReadDir(dirname string) ([]fs.DirEntry, error)
    ReadFile(filename string) ([]byte, error)
    Readlink(name string) (string, error)
    Remove(name string) error
    RemoveAll(name string) error
    Rename(oldpath, newpath string) error
    Stat(name string) (fs.FileInfo, error)
    Symlink(oldname, newname string) error
    Truncate(name string, size int64) error
    WriteFile(filename string, data []byte, perm fs.FileMode) error
}

To use vfs, you write your code to use the FS interface, and then use vfst to test it.

vfs also provides functions MkdirAll (equivalent to os.MkdirAll), Contains (an improved filepath.HasPrefix), and Walk (equivalent to filepath.Walk) that operate on an FS.

The implementations of FS provided are:

• OSFS which calls the underlying os and io functions directly.

• PathFS which transforms all paths to provide a poor-man's chroot.

• ReadOnlyFS which prevents modification of the underlying FS.

• TestFS which assists running tests on a real filesystem but in a temporary directory that is easily cleaned up. It uses OSFS under the hood.

Example usage:

// writeConfigFile is the function we're going to test. It can make arbitrary
// changes to the filesystem through fileSystem.
func writeConfigFile(fileSystem vfs.FS) error {
    return fileSystem.WriteFile("/home/user/app.conf", []byte(`app config`), 0644)
}

// TestWriteConfigFile is our test function.
func TestWriteConfigFile(t *testing.T) {
    // Create and populate an temporary directory with a home directory.
    fileSystem, cleanup, err := vfst.NewTestFS(map[string]any{
        "/home/user/.bashrc": "# contents of user's .bashrc\n",
    })

    // Check that the directory was populated successfully.
    if err != nil {
        t.Fatalf("vfsTest.NewTestFS(_) == _, _, %v, want _, _, <nil>", err)
    }

    // Ensure that the temporary directory is removed.
    defer cleanup()

    // Call the function we want to test.
    if err := writeConfigFile(fileSystem); err != nil {
        t.Error(err)
    }

    // Check properties of the filesystem after our function has modified it.
    vfst.RunTests(t, fileSystem, "app_conf",
        vfst.TestPath("/home/user/app.conf",
            vfst.TestModeIsRegular(),
            vfst.TestModePerm(0644),
            vfst.TestContentsString("app config"),
        ),
    )
}

github.com/spf13/afero compatibility

There is a compatibility shim for github.com/spf13/afero in github.com/twpayne/go-vfsafero. This allows you to use vfst to test existing code that uses afero.FS. See the documentation for an example.

github.com/src-d/go-billy compatibility

There is a compatibility shim for github.com/src-d/go-billy in github.com/twpayne/go-vfsbilly. This allows you to use vfst to test existing code that uses billy.Filesystem. See the documentation for an example.

Motivation

vfs was inspired by github.com/spf13/afero. So, why not use afero?

• afero has several critical bugs in its in-memory mock filesystem implementation MemMapFs, to the point that it is unusable for non-trivial test cases. vfs does not attempt to implement an in-memory mock filesystem, and instead only provides a thin layer around the standard library's os and io packages, and as such should have fewer bugs.

• afero does not support creating or reading symbolic links, and its LstatIfPossible interface is clumsy to use as it is not part of the afero.Fs interface. vfs provides out-of-the-box support for symbolic links with all methods in the FS interface.

• afero has been effectively abandoned by its author, and a "friendly fork" (github.com/absfs/afero) has not seen much activity. vfs, by providing much less functionality than afero, should be smaller and easier to maintain.

License

MIT

go-vfs's Issues

PathFS should prevent writes outside its root

Current PathFS uses filepath.Join to combine names with the root. This does not prevent writes outside the root, for example by using relative paths. PathFS should prevent writes outside its root.

Additional FS

Hey! Nice work... Golang really needs a standard Filesystem interface upstream, or at least one that everyone gathers around. In any case, maybe my etcdfs work is helpful here. I'm not contributing to afero anymore because: spf13/afero#184

I saw your compatibility layer in https://github.com/twpayne/go-vfsafero -- it would be interesting to be able to go the other way!

In any case, feel free to close this, thought it might be a good place for discussing additional FS.

Minio/S3 support

Thanks for the great repository. Do you have any plan to support Minio/S3 support? Maybe you could use minio's SDK to do that.

Improve Windows support

While working on twpayne/chezmoi#23 I ran the go-vfs tests on Windows and encountered a couple of failures. I've put together some fixes (https://github.com/zb140/go-vfs/tree/windows) after which all of the tests pass on both Windows and Linux (I don't have a Mac so I haven't tested there).

There were a couple of changes necessary that I think are worth calling special attention to:

• On Windows, symlinks should not be created before the things they point to exist (because links to directories are different from links to files, and the difference must be explicitly specified when calling CreateSymbolicLink). Since the filesystems in vfst are declared as a map and you can't rely on iteration order, tests that were creating symlinks to directories were often creating them before the directories existed, causing them not to work. I solved this by adding support to Builder.build to be able to take a list of any of the types it previously supported, then recursively call them in order. See https://github.com/zb140/go-vfs/blob/cd3b3edf4f462bb963afdab8900eb7e6836fff33/vfst/vfst.go#L83
• When Windows is asked to make a directory containing a path component that exists but is not a directory, the Windows APIs return (the equivalent of) ENOENT rather than ENOTDIR. This causes MkdirAll to attempt to recursively create the path, which will fail because it already exists. This is detected correctly but MkdirAll was swallowing the error, causing the test to detect no error when one was expected. I fixed it by returning the error rather than ignoring it. See https://github.com/zb140/go-vfs/blob/cd3b3edf4f462bb963afdab8900eb7e6836fff33/mkdirall.go#L49
• PathFS.join expects incoming path names to be absolute before the internal prefix is added. But on Windows, a path is never considered absolute if it doesn't start with a volume specifier, which means the check was always failing. To fix this, I coerce the incoming paths to Unix-style using filepath.ToSlash, then check for an absolute path using path.IsAbs instead of filepath.IsAbs. See https://github.com/zb140/go-vfs/blob/cd3b3edf4f462bb963afdab8900eb7e6836fff33/path_fs.go#L224