spf13 / afero

A FileSystem Abstraction System for Go

@spf13 👋

Former:

afero's Issues

How to vendor our dependencies?

@spf13 What process should we use to vendor our dependencies?

Now that Go 1.6 is released and vendoring is enabled by default we should use something that uses the /vendor subdirectory approach. Should we use a tool for that or just make a subtree merge with squashed commits?

The reason why we should vendor is for example this issue: #59

Union mount

Is it possible to use some sort of union mount with this library?

syscall.EBADFD at FreeBSD

Hello,
I was trying to use afero as Hugo dependency at FreeBSD:

$ go get github.com/spf13/afero
# github.com/spf13/afero
gocode/src/github.com/spf13/afero/const_win_unix.go:22: undefined: syscall.EBADFD

$ uname -a
FreeBSD nixbsd 10.2-RELEASE-p9 FreeBSD 10.2-RELEASE-p9 #0: Thu Jan 14 01:32:46 UTC 2016     [email protected]:/usr/obj/usr/src/sys/GENERIC  amd64
$ go version
go version go1.5.3 freebsd/amd64

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOOS="freebsd"
GOPATH="/home/nixusr/gocode"
GORACE=""
GOROOT="/home/nixusr/go"
GOTOOLDIR="/home/nixusr/go/pkg/tool/freebsd_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc48"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
$

Then, is FreeBSD not supported yet?

Sftp_test has errors

./sftp_test.go:203: cannot use debugStream (type io.Writer) as type string in argument to sftp.NewServer
./sftp_test.go:203: cannot use debugLevel (type int) as type sftp.ServerOption in argument to sftp.NewServer
./sftp_test.go:203: cannot use readOnly (type bool) as type sftp.ServerOption in argument to sftp.NewServer
./sftp_test.go:203: cannot use rootpath (type string) as type sftp.ServerOption in argument to sftp.NewServer

I'm not sure if it's just me, but this doesn't look like it should have ever built. If I'm doing something wrong, please let me know.

MemMapFs should return errors as os.PathError

The os package always returns a os.PathError value as error on filesystem operations. This gives additional context that is helpful for the user.

type PathError struct {
        Op   string
        Path string
        Err  error
}

Current error output is: file does not exist
PathError output would look like that: open /some/path: file does not exist

MemFile does not implement io.WriteTo

What would be the best way to convert a mem file to another backend? I was thinking io.copy? Won't mem file need to implement WriteTo?

move fs implementations into sub packages

Right now if I depend on github.com/spf13/afero, I also have to pull in a lot of potentially unnecessary dependencies that might not be related to the FS implementation I'm using, e.g. github.com/pkg/sftp.

Let me know if you agree and would be open to a PR.

release

Is this library stable ? Just noticed that there have not been any releases yet and so thought i should ask.

I actually need the SFTP functionality.

Regression in Hugo

Hugo builds started to fail recently due to an issue with Afero:

links
--- PASS: TestSiteInfoPermalinks (0.00s)
=== RUN TestSitemapOutput
--- FAIL: TestSitemapOutput (0.01s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0x660933]

goroutine 3538 [running]:
testing.func·006()
    c:/go/src/testing/testing.go:441 +0x188
github.com/spf13/afero.(*MemMapFs).registerWithParent(0xc082582fa0, 0x1679a0, 0xc082103c70)
    C:/GOPATH/src/github.com/spf13/afero/memmap.go:143 +0x463
github.com/spf13/afero.(*MemMapFs).Create(0xc082582fa0, 0xc0828368ba, 0x1, 0x0, 0x0, 0x0, 0x0)
    C:/GOPATH/src/github.com/spf13/afero/memmap.go:93 +0x19a
github.com/spf13/hugo/helpers.WriteToDisk(0xc0828368ba, 0x1, 0x167b60, 0xc082367500, 0x162ca0, 0xc082582fa0, 0x0, 0x0)
    C:/GOPATH/src/github.com/spf13/hugo/helpers/path.go:555 +0x198
github.com/spf13/hugo/target.(*Filesystem).Publish(0xc08232a5c0, 0x0, 0x0, 0x167b60, 0xc082367500, 0x0, 0x0)
    C:/GOPATH/src/github.com/spf13/hugo/target/file.go:39 +0xce
github.com/spf13/hugo/hugolib.(*Site).WriteDestFile(0xc082864d20, 0x0, 0x0, 0x167b60, 0xc082367500, 0x0, 0x0)
    github.com/spf13/hugo/hugolib/_test/_obj_test/site.go:1946 +0x1e8
github.com/spf13/hugo/hugolib.(*Site).renderAndWriteXML(0xc082864d20, 0xb4def0, 0xc, 0x0, 0x0, 0xace0c0, 0xc0827c8c60, 0xc0827cb290, 0x3, 0x3, ...)
    github.com/spf13/hugo/hugolib/_test/_obj_test/site.go:1772 +0x40a
github.com/spf13/hugo/hugolib.(*Site).RenderHomePage(0xc082864d20, 0x0, 0x0)
    github.com/spf13/hugo/hugolib/_test/_obj_test/site.go:1621 +0x83f
github.com/spf13/hugo/hugolib.TestSitemapOutput(0xc0825ea090)
    C:/GOPATH/src/github.com/spf13/hugo/hugolib/sitemap_test.go:50 +0x477
testing.tRunner(0xc0825ea090, 0xe86160)
    c:/go/src/testing/testing.go:447 +0xc6
created by testing.RunTests
    c:/go/src/testing/testing.go:555 +0xa92

goroutine 1 [chan receive]:
testing.RunTests(0xc6cbc8, 0xe85440, 0x8e, 0x8e, 0xe88f01)
    c:/go/src/testing/

Haven't looked closer.

File not found when parent directory not present

In copyOnWriteFs.go you test if the file is not found in the layer Fs, but if the parent directory is not present either, you will get syscall.ENOTDIR instead of syscall.ENOENT. This only happens on Windows, you can find background information here: https://codereview.appspot.com/4984051/

A solution could be to replace line 35 with:
if oerr.Err == os.ErrNotExist || oerr.Err == syscall.ENOENT || oerr.Err == syscall.ENOTDIR {

and change the test at line 83:

if err == syscall.ENOENT || err == syscall.ENOTDIR {
    return u.base.Stat(name)
}

This should fix the failing unit test and solve an issue in Hugo where CSS is not copied from the theme if your site static directory does not already contain a css directory.

MemMapFS: uses shared file position

EDITed: fixed typo
The following code demonstrates this behavior, it should print

    OS=<datadata>
    FS=<datadata>

but the FS line reads FS=<testdatadata>

package main

import (
    "os"
    "fmt"
    "io/ioutil"
    "github.com/spf13/afero"
)

func main() {
    fileName := "./afero-demo2.txt"
    fh1, err := os.Create(fileName)
    if err != nil {
        panic("os.Create failed: "+err.Error())
    }
    _, err = fh1.Write([]byte("test"))
    if err != nil {
        panic("fh.Write failed: "+err.Error())
    }
    fh1.Seek(0, os.SEEK_SET)

    fh2, err := os.OpenFile(fileName, os.O_RDWR, 0777)
    if err != nil {
        panic("os.OpenFile failed: "+err.Error())
    }
    fh2.Seek(0, os.SEEK_END)
    fh2.Write([]byte("data"))
    fh2.Close()

    fh1.Write([]byte("data"))
    fh1.Close()
    // the file now should contain "datadata"
    data, _ := ioutil.ReadFile(fileName)
    fmt.Printf("OS=<%s>\n", data)

    fs := &afero.MemMapFs{}
    afh1, err := fs.Create(fileName)
    if err != nil {
        panic("fs.Create failed: "+err.Error())
    }
    _, err = afh1.Write([]byte("test"))
    if err != nil {
        panic("fh.Write failed: "+err.Error())
    }
    afh1.Seek(0, os.SEEK_SET)

    afh2, err := fs.OpenFile(fileName, os.O_RDWR, 0777)
    if err != nil {
        panic("fs.OpenFile failed: "+err.Error())
    }
    afh2.Seek(0, os.SEEK_END)
    afh2.Write([]byte("data"))
    afh2.Close()

    afh1.Write([]byte("data"))
    afh1.Close()
    // the file now should contain "datadata"
    data, _ = afero.ReadFile(fs, fileName)
    fmt.Printf("FS=<%s>\n", data)
}

Greek translation

In modern Greek, aferimenos (αφηρημένος) means abstract (as an adjective), and afairo (αφαιρώ) is pronounced like "afero", but it means to subtract.

MemMapFS: opening a file read-only should fail on writes

The following code panics on line 47 (at afh.Write()), because it can write to the read-only opened file handle:

package main

import (
    "os"
    "github.com/spf13/afero"
)

func main() {
    fileName := "./afero-demo.txt"
    fh, err := os.Create(fileName)
    if err != nil {
        panic("os.Create failed: "+err.Error())
    }
    _, err = fh.Write([]byte("test"))
    if err != nil {
        panic("fh.Write failed: "+err.Error())
    }
    fh.Close()

    fh, err = os.Open(fileName)
    if err != nil {
        panic("os.Open failed: "+err.Error())
    }
    _, err = fh.Write([]byte("data"))
    if err == nil {
        panic("No write error")
    }
    fh.Close()

    fs := &afero.MemMapFs{}
    afh, err := fs.Create(fileName)
    if err != nil {
        panic("fs.Create failed: "+err.Error())
    }
    _, err = afh.Write([]byte("test"))
    if err != nil {
        panic("fh.Write failed: "+err.Error())
    }
    afh.Close()

    afh, err = fs.Open(fileName)
    if err != nil {
        panic("fs.Open failed: "+err.Error())
    }
    _, err = afh.Write([]byte("data"))
    if err == nil {
        panic("No write error")
    }
    afh.Close()
}

Unable to "chain" BasePathFs

This example does not work as expected:

package main

import (
    "fmt"

    "github.com/spf13/afero"
)

func main() {
    fs1 := afero.NewBasePathFs(afero.NewOsFs(), "/dir1/subdir1")
    fmt.Println(fs1.(*afero.BasePathFs).RealPath("file.txt"))
    fs2 := afero.NewBasePathFs(fs1, "/dir2/subdir2")
    fmt.Println(fs2.(*afero.BasePathFs).RealPath("file.txt"))
}

The above output is:

/dir1/subdir1/file.txt
/dir2/subdir2/file.txt

Instead I expected fs2 to have nested paths as it uses fs1 as its source:

/dir1/subdir1/file.txt
/dir1/subdir1/dir2/subdir2/file.txt

Issue to add images to readme

example for HttpFs

It would be nice to have example of usage of HttpFs, I was wondering if it could be used to fetch content from github to feed into hugo.

Abstracting S3

It would be useful to abstract s3 http like file systems too.

I have been using Mino. It's really cool because it has 3 different avatar tions itself:
Amazon S3, Google Storage, and its own Minio server.

The minio server is cool because it does dedup and some other smarts too.

So,my suggestion is to incorporate this, so it works with the existing API.

What do others think ?

Tachyon backend

Afero is an awesome abstraction for file system. Tachyon is a memory first file system backed with a storage backend.
It still doesn't have a Go client and afero could work quite well with it.
What do you think?

"go test" fails on Windows with current Afero HEAD

As of today, e.g. at commit a5d3e7c, go test on Afero fails these two tests on Windows:

• TestNestedDirOverlayOsFsReaddir
• TestCopyOnWriteFsWithOsFs

Console log:

Veronica@Veronica-V5 MINGW64 ~/GoPath/src/github.com/spf13/afero (master)
$ go test ./...
error removing tempDir C:\Users\Veronica\AppData\Local\Temp\625740957 remove C:\Users\Veronica\AppData\Local\Temp\625740957\home\test: The directory is not empty.
--- FAIL: TestNestedDirOverlayOsFsReaddir (0.01s)
        composite_test.go:45: error cleaning up tempDirs
error removing tempDir C:\Users\Veronica\AppData\Local\Temp\054838104 remove C:\Users\Veronica\AppData\Local\Temp\054838104\home: The directory is not empty.
error removing tempDir C:\Users\Veronica\AppData\Local\Temp\512058071 remove C:\Users\Veronica\AppData\Local\Temp\512058071\home: The directory is not empty.
--- FAIL: TestCopyOnWriteFsWithOsFs (0.01s)
        composite_test.go:45: error cleaning up tempDirs
FAIL
FAIL    github.com/spf13/afero  6.496s
?       github.com/spf13/afero/mem      [no test files]
?       github.com/spf13/afero/sftp     [no test files]

Console log in verbose test mode:

Veronica@Veronica-V5 MINGW64 ~/GoPath/src/github.com/spf13/afero (master)
$ go test -v ./...
=== RUN   TestRead0
--- PASS: TestRead0 (0.00s)
=== RUN   TestOpenFile
--- PASS: TestOpenFile (0.00s)
=== RUN   TestCreate
--- PASS: TestCreate (0.00s)
=== RUN   TestMemFileRead
--- PASS: TestMemFileRead (0.00s)
=== RUN   TestRename
--- PASS: TestRename (0.00s)
=== RUN   TestRemove
--- PASS: TestRemove (0.01s)
=== RUN   TestTruncate
--- PASS: TestTruncate (0.00s)
=== RUN   TestSeek
--- PASS: TestSeek (0.00s)
=== RUN   TestReadAt
--- PASS: TestReadAt (0.00s)
=== RUN   TestWriteAt
--- PASS: TestWriteAt (0.00s)
=== RUN   TestReaddirnames
--- PASS: TestReaddirnames (0.01s)
=== RUN   TestReaddirSimple
--- PASS: TestReaddirSimple (0.01s)
=== RUN   TestReaddir
--- PASS: TestReaddir (0.10s)
=== RUN   TestReaddirAll
--- PASS: TestReaddirAll (0.01s)
=== RUN   TestBasePath
--- PASS: TestBasePath (0.00s)
=== RUN   TestBasePathRoot
--- PASS: TestBasePathRoot (0.00s)
=== RUN   TestUnionCreateExisting
--- PASS: TestUnionCreateExisting (0.00s)
=== RUN   TestUnionMergeReaddir
--- PASS: TestUnionMergeReaddir (0.00s)
=== RUN   TestExistingDirectoryCollisionReaddir
--- PASS: TestExistingDirectoryCollisionReaddir (0.00s)
=== RUN   TestNestedDirBaseReaddir
--- PASS: TestNestedDirBaseReaddir (0.00s)
=== RUN   TestNestedDirOverlayReaddir
--- PASS: TestNestedDirOverlayReaddir (0.00s)
=== RUN   TestNestedDirOverlayOsFsReaddir
error removing tempDir C:\Users\Veronica\AppData\Local\Temp\322120829 remove C:\Users\Veronica\AppData\Local\Temp\322120829\home\test: The directory is not empty.
--- FAIL: TestNestedDirOverlayOsFsReaddir (0.01s)
        composite_test.go:45: error cleaning up tempDirs
=== RUN   TestCopyOnWriteFsWithOsFs
error removing tempDir C:\Users\Veronica\AppData\Local\Temp\599030712 remove C:\Users\Veronica\AppData\Local\Temp\599030712\home: The directory is not empty.
error removing tempDir C:\Users\Veronica\AppData\Local\Temp\016726455 remove C:\Users\Veronica\AppData\Local\Temp\016726455\home: The directory is not empty.
--- FAIL: TestCopyOnWriteFsWithOsFs (0.01s)
        composite_test.go:45: error cleaning up tempDirs
=== RUN   TestUnionCacheWrite
--- PASS: TestUnionCacheWrite (0.00s)
=== RUN   TestUnionCacheExpire
--- PASS: TestUnionCacheExpire (2.00s)
=== RUN   TestReadFile
--- PASS: TestReadFile (0.00s)
=== RUN   TestWriteFile
--- PASS: TestWriteFile (0.00s)
=== RUN   TestReadDir
--- PASS: TestReadDir (0.00s)
=== RUN   TestNormalizePath
--- PASS: TestNormalizePath (0.00s)
=== RUN   TestPathErrors
--- PASS: TestPathErrors (0.00s)
=== RUN   TestMultipleOpenFiles
--- PASS: TestMultipleOpenFiles (0.01s)
=== RUN   TestReadOnly
--- PASS: TestReadOnly (0.00s)
=== RUN   TestWriteCloseTime
--- PASS: TestWriteCloseTime (4.01s)
=== RUN   TestWalk
--- PASS: TestWalk (0.03s)
=== RUN   TestFilterReadOnly
--- PASS: TestFilterReadOnly (0.00s)
=== RUN   TestFilterReadonlyRemoveAndRead
--- PASS: TestFilterReadonlyRemoveAndRead (0.00s)
=== RUN   TestFilterRegexp
--- PASS: TestFilterRegexp (0.00s)
=== RUN   TestFilterRORegexpChain
--- PASS: TestFilterRORegexpChain (0.00s)
=== RUN   TestFilterRegexReadDir
--- PASS: TestFilterRegexReadDir (0.00s)
=== RUN   TestDirExists
--- PASS: TestDirExists (0.00s)
=== RUN   TestIsDir
--- PASS: TestIsDir (0.00s)
=== RUN   TestIsEmpty
--- PASS: TestIsEmpty (0.00s)
=== RUN   TestExists
--- PASS: TestExists (0.00s)
=== RUN   TestSafeWriteToDisk
--- PASS: TestSafeWriteToDisk (0.00s)
=== RUN   TestWriteToDisk
--- PASS: TestWriteToDisk (0.00s)
=== RUN   TestGetTempDir
--- PASS: TestGetTempDir (0.00s)
FAIL
FAIL    github.com/spf13/afero  6.530s
?       github.com/spf13/afero/mem      [no test files]
?       github.com/spf13/afero/sftp     [no test files]

Internalize some types used by MemMapFs

I was just looking at afero in godoc and wondered about the exported types that are probably not used directly by users of the library as they only exist to support MemMapFs. I propose to unexport the following types for the sake of simplifying the API and focusing the documentation on the important parts of afero:

• type InMemoryFile
• type InMemoryFileInfo
• type MemDir
• type MemDirMap

Do you think that this is a good idea, or do you know any projects that use these types?

Explain the difference between filter and filesystem better

@vetinari The recent additions you've provided are great. Well written and thoughtfully done. In reading through the readme, it's not at all apparent what the difference is between a filter and a filesystem considering both adhere to the same interface.

As I see it the main difference is that the filter is stackable at call time where any combination of filesystems as a filesystem needs to be predefined. It does beg the question though if everything should be a filter (or if we should incorporate that functionality into the FS since that's a compelling feature.

If you could elaborate on the readme I'm sure many would benefit.

One-way encrypted file system

Seems like a good fit for afero, ala:

http://owefs.firelet.net/

MemMapFs is racy

go test -race ./... in Hugo:

WARNING: DATA RACE
Read by goroutine 57:
  github.com/spf13/afero.(*MemMapFs).Mkdir()
      /Users/bep/go/src/github.com/spf13/afero/memmap.go:156 +0xa3
  github.com/spf13/afero.(*MemMapFs).MkdirAll()
      /Users/bep/go/src/github.com/spf13/afero/memmap.go:178 +0x56
  github.com/spf13/afero.WriteReader()
      /Users/bep/go/src/github.com/spf13/afero/util.go:46 +0xec
  github.com/spf13/hugo/helpers.WriteToDisk()
      /Users/bep/go/src/github.com/spf13/hugo/helpers/path.go:438 +0x7b
  github.com/spf13/hugo/target.(*PagePub).Publish()
      /Users/bep/go/src/github.com/spf13/hugo/target/page.go:44 +0xf2
  github.com/spf13/hugo/hugolib.(*Site).WriteDestPage()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1606 +0x257
  github.com/spf13/hugo/hugolib.(*Site).renderAndWritePage()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1516 +0x1142
  github.com/spf13/hugo/hugolib.pageRenderer()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:974 +0x25a

Previous write by goroutine 65:
  github.com/spf13/afero.(*MemMapFs).Mkdir()
      /Users/bep/go/src/github.com/spf13/afero/memmap.go:156 +0xfa
  github.com/spf13/afero.(*MemMapFs).MkdirAll()
      /Users/bep/go/src/github.com/spf13/afero/memmap.go:178 +0x56
  github.com/spf13/afero.WriteReader()
      /Users/bep/go/src/github.com/spf13/afero/util.go:46 +0xec
  github.com/spf13/hugo/helpers.WriteToDisk()
      /Users/bep/go/src/github.com/spf13/hugo/helpers/path.go:438 +0x7b
  github.com/spf13/hugo/target.(*PagePub).Publish()
      /Users/bep/go/src/github.com/spf13/hugo/target/page.go:44 +0xf2
  github.com/spf13/hugo/hugolib.(*Site).WriteDestPage()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1606 +0x257
  github.com/spf13/hugo/hugolib.(*Site).renderAndWritePage()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1516 +0x1142
  github.com/spf13/hugo/hugolib.pageRenderer()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:974 +0x25a

WARNING: DATA RACE
Read by goroutine 57:
  runtime.mapaccess2_faststr()
      /private/var/folders/vd/7l9ys5k57l91x63sh28wl_kc0000gn/T/workdir/go/src/runtime/hashmap_fast.go:281 +0x0
  github.com/spf13/afero.(*MemMapFs).Mkdir()
      /Users/bep/go/src/github.com/spf13/afero/memmap.go:156 +0x3a3
  github.com/spf13/afero.(*MemMapFs).MkdirAll()
      /Users/bep/go/src/github.com/spf13/afero/memmap.go:178 +0x56
  github.com/spf13/afero.WriteReader()
      /Users/bep/go/src/github.com/spf13/afero/util.go:46 +0xec
  github.com/spf13/hugo/helpers.WriteToDisk()
      /Users/bep/go/src/github.com/spf13/hugo/helpers/path.go:438 +0x7b
  github.com/spf13/hugo/target.(*PagePub).Publish()
      /Users/bep/go/src/github.com/spf13/hugo/target/page.go:44 +0xf2
  github.com/spf13/hugo/hugolib.(*Site).WriteDestPage()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1606 +0x257
  github.com/spf13/hugo/hugolib.(*Site).renderAndWritePage()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1516 +0x1142
  github.com/spf13/hugo/hugolib.pageRenderer()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:974 +0x25a

Previous write by goroutine 65:
  runtime.mapassign1()
      /private/var/folders/vd/7l9ys5k57l91x63sh28wl_kc0000gn/T/workdir/go/src/runtime/hashmap.go:411 +0x0
  github.com/spf13/afero.(*MemMapFs).Mkdir()
      /Users/bep/go/src/github.com/spf13/afero/memmap.go:156 +0x34e
  github.com/spf13/afero.(*MemMapFs).MkdirAll()
      /Users/bep/go/src/github.com/spf13/afero/memmap.go:178 +0x56
  github.com/spf13/afero.WriteReader()
      /Users/bep/go/src/github.com/spf13/afero/util.go:46 +0xec
  github.com/spf13/hugo/helpers.WriteToDisk()
      /Users/bep/go/src/github.com/spf13/hugo/helpers/path.go:438 +0x7b
  github.com/spf13/hugo/target.(*PagePub).Publish()
      /Users/bep/go/src/github.com/spf13/hugo/target/page.go:44 +0xf2
  github.com/spf13/hugo/hugolib.(*Site).WriteDestPage()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1606 +0x257
  github.com/spf13/hugo/hugolib.(*Site).renderAndWritePage()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1516 +0x1142
  github.com/spf13/hugo/hugolib.pageRenderer()
      /Users/bep/go/src/github.com/spf13/hugo/hugolib/site
``

Renaming const_darwin.go to const_darwin_openbsd.go causes other applications to fail

The rename of const_darwin.go seems to break the Hugo build on OS X:

$ cd spf13/hugo
$ go build
# github.com/spf13/afero
../afero/unionFile.go:40: undefined: BADFD
../afero/unionFile.go:60: undefined: BADFD
../afero/unionFile.go:74: undefined: BADFD
../afero/unionFile.go:88: undefined: BADFD
../afero/unionFile.go:102: undefined: BADFD
../afero/unionFile.go:116: undefined: BADFD
../afero/unionFile.go:183: undefined: BADFD
../afero/unionFile.go:197: undefined: BADFD
../afero/unionFile.go:211: undefined: BADFD
../afero/unionFile.go:225: undefined: BADFD
../afero/unionFile.go:225: too many errors

Copying it back seems to fix it:

$ cp ../afero/const_darwin_openbsd.go ../afero/const_darwin.go
$ go build
$ 

CopyOnWriteFs fails to CreateFile

I was getting stat IsNotExist errors when doing a MkdirAll followed by a an OpenFile. It seemed that that CopyOnWriteFs was checking the base os first to see if the base directory of the opened file IsDir, but that led to a stat error being returned.

line in question:
https://github.com/spf13/afero/blob/master/copyOnWriteFs.go#L149

my quick solution.

        dir := filepath.Dir(name)
        var isaDir bool
        if isaDir, err = IsDir(u.base, dir); os.IsNotExist(err) {
            // skip ahead
        } else if err != nil {
            return nil, err
        }
        if isaDir {
            if err = u.layer.MkdirAll(dir, 0777); err != nil {
                return nil, err
            }
            return u.layer.OpenFile(name, flag, perm)
        }  

Symlink handling in BasePathFs

I would like to use Afero's BasePathFs to simluate a chroot-like environment any I was unsure how you'd handle symlinks.

For example, if I have bin/foo Symlinks to /usr/bin/bar and then I put both these files in /tmp/afero-test and try to run this:

bp := afero.NewBasePathFs(afero.NewOsFs(), "/tmp/afero-test")
fh, err := bp.Open("/bin/foo")

Do i get an error? (I think right now I do) I'd kind of like this to "rewrite" the symlinks so that I actually open /tmp/afero-test/usr/bin/bar. Would this be an acceptable change to make?

(Is supporting symlinks in one type of FS but not others going to cause problems?)

Afero can't handle relative paths properly

This code:

package main

import (
    "fmt"
    "os"
    "path/filepath"

    "github.com/spf13/afero"
)

func main() {
    fs := new(afero.MemMapFs)

    f, _ := fs.Create(filepath.Join(".", "file"))
    f.WriteString("TestFile content")
    f.Close()

    wd, _ := os.Getwd()

    _, err := fs.Open(filepath.Join(wd, "file"))
    if err != nil {
        fmt.Println(err)
    }

    fmt.Println("wd:", wd)

    fs.List()
}

Produces this output:

$ absrel
file does not exist
wd: /tmp
file 16
. 42

It doesn't make sense to save relative paths into the MemMapFs, they all should be absolute.

I am working on a fix for that problem, but I am not sure if this can work together with the working directory from the os package. The problem is that if I use os.Chdir("/tmp/afero"), and then create a file "./file" on OS X it is actually put into "/private/tmp/afero". So if I use the dir string that I used in the Chdir function, the file is also not found. I also can't always use filepath.EvalSymlinks because files that exist in a MemMapFs won't exist in the real file system.

MemMapFs panics sometimes

From Hugo tests:

fatal error: concurrent map writes

goroutine 116 [running]:
runtime.throw(0x888c90, 0x15)
    /Users/bep/dev/clone/go/src/runtime/panic.go:530 +0x90 fp=exit status 2
FAIL    github.com/spf13/hugo/hugolib   0.031s
0xc82003e988 sp=0xc82003e970
runtime.mapassign1(0x630940, 0xc8200a33e0, 0xc82003eb38, 0xc82003eb28)
    /Users/bep/dev/clone/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc82003ea30 sp=0xc82003e988
github.com/spf13/afero.(*MemMapFs).Mkdir(0xc82018b570, 0xc8200aa800, 0x4, 0xc8000001ff, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/afero/memmap.go:156 +0x297 fp=0xc82003eb50 sp=0xc82003ea30
github.com/spf13/afero.(*MemMapFs).MkdirAll(0xc82018b570, 0xc8200aa800, 0x5, 0x1ff, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/afero/memmap.go:178 +0x49 fp=0xc82003eb88 sp=0xc82003eb50
github.com/spf13/afero.WriteReader(0xdb8130, 0xc82018b570, 0xc8200aa800, 0xd, 0xdb2a60, 0xc8203b4150, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/afero/util.go:46 +0xd5 fp=0xc82003ec30 sp=0xc82003eb88
github.com/spf13/hugo/helpers.WriteToDisk(0xc8200aa800, 0xd, 0xdb2a60, 0xc8203b4150, 0xdb8130, 0xc82018b570, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/hugo/helpers/path.go:438 +0x64 fp=0xc82003ec78 sp=0xc82003ec30
github.com/spf13/hugo/target.(*Filesystem).Publish(0xc8203cca60, 0x830a00, 0xd, 0xdb2a60, 0xc8203b4150, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/hugo/target/file.go:52 +0xc0 fp=0xc82003ecd0 sp=0xc82003ec78
github.com/spf13/hugo/hugolib.(*Site).WriteDestFile(0xc8201a4000, 0x830a00, 0xd, 0xdb2a60, 0xc8203b4150, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1601 +0x1ee fp=0xc82003ed88 sp=0xc82003ecd0
github.com/spf13/hugo/hugolib.defaultHandler.FileConvert(0x0, 0x0, 0x0, 0xc8201b4af0, 0xc8201a4000, 0x0, 0x0, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/hugo/hugolib/handler_file.go:42 +0x83 fp=0xc82003edd8 sp=0xc82003ed88
github.com/spf13/hugo/hugolib.(*defaultHandler).FileConvert(0xc8201a6fa0, 0xc8201b4af0, 0xc8201a4000, 0x0, 0x0, 0x0, 0x0)
    <autogenerated>:99 +0xce fp=0xc82003ee28 sp=0xc82003edd8
github.com/spf13/hugo/hugolib.(*MetaHandle).Convert(0xc8201a7da0, 0x7cdae0, 0xc8201b4af0, 0xc8201a4000, 0xc8203e6300)
    /Users/bep/go/src/github.com/spf13/hugo/hugolib/handler_meta.go:62 +0x8e fp=0xc82003ef28 sp=0xc82003ee28
github.com/spf13/hugo/hugolib.fileConverter(0xc8201a4000, 0xc8203e63c0, 0xc8203e6300, 0xc8203cc430)
    /Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:640 +0x107 fp=0xc82003ef90 sp=0xc82003ef28
runtime.goexit()
    /Users/bep/dev/clone/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82003ef98 sp=0xc82003ef90
created by github.com/spf13/hugo/hugolib.(*Site).CreatePages
    /

Initial sftp backend draft

Hi guys,

I'm fairly new with golang and trying to get my hands dirty and have got a initial working draft on a SFTP backend.

What works

• Connecting/disconnecting (with $HOME/.ssh/id_rsa)
• Creating and writing a file
• Most trivial functions

What works not

• Passing advanced SSH/sftp config

Here is a example:

        fs := new(afero.SftpFs)

        fs.Connect("user", "host:port")

        fs.Mkdir("foo", os.FileMode(0000))
        fs.Mkdir("bar", os.FileMode(0777))

        f, err := fs.Create("dusse")
        if err != nil {
                log.Fatalf("open: %v", err)
        }
        defer f.Close()

        f.Write([]byte("awesomeeeeee"))

        fs.MkdirAll("dir1/dir2/dir3", os.FileMode(0777))

        fs.Remove("dir1/dir2/dir3")
        fs.Remove("dir1/dir2")

        fs.Disconnect()

The thing lives currently in my forked branch to have some talking before I'm creating a pull request because I felt it is not ready to be merged yet.

https://github.com/xor-gate/afero/blob/sftp/sftp.go
https://github.com/xor-gate/afero/blob/sftp/sftp/file.go

Kind regards,
Jerry

Data race in InMemoryFile

Read and Write called from concurrent goroutines causes a data race on the InMemoryFile file contents. A simple fix is to lock access to f.data in both methods, by including an anonymous sync.Mutex in the InMemoryFile struct.

OsFs test files not deleted on Windows

When running tests on Windows (go 1.5.1 amd64), all of the tests pass, but the test files are not removed as expected.

I'll submit a PR shortly to address this issue.

Never log.Fatal

Which does a os.Exit

See

afero/util.go

Line 49 in e042b5f

log.Fatalln(err)

This is not good behavior for a library. Replace these with a log.Panic.

New*Fs functions should return the concrete type

Having to do:

 NewBasePathFs(fs, baseDir).(*BasePathFs)

Doesn't make much sense. I don't see the problem going the other way around; but as this is a file system with a very specific contract, I want to know what it is.

Close()ing a MemMapFs file opened R/W should set mod time

Currently only files Create()d get their modification time changed, i.e. OpenFile() with flags os.O_RDRW / os.O_WRONLY never changes the modification time of a file

BasePathFs.RealPath panics on empty base path

An ErrNotExist would be better, maybe -- or maybe return an error on construction. This sounds like a bug in Go's stdlib, I'll report an issue. Not sure what an empty file path is supposed to work as.

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x1cf9af]

goroutine 227 [running]:
panic(0x723440, 0xc82000a1b0)
    /usr/local/go/src/runtime/panic.go:464 +0x3e6
testing.tRunner.func1(0xc8202e93b0)
    /usr/local/go/src/testing/testing.go:467 +0x192
panic(0x723440, 0xc82000a1b0)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
github.com/spf13/afero.(*BasePathFs).RealPath(0x0, 0x7f3a68, 0x1, 0x0, 0x0, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/afero/basepath.go:37 +0x2ff
github.com/spf13/afero.(*BasePathFs).Open(0x0, 0x7f3a68, 0x1, 0x0, 0x0, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/afero/basepath.go:118 +0x5b
github.com/spf13/afero.ReadFile(0x18c42c0, 0x0, 0x7f3a68, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/bep/go/src/github.com/spf13/afero/ioutil.go:65 +0x7f
``

InMemFile.Read has inconsistent behaviour at EOF with (*os.File).Read

http://golang.org/pkg/os/#File.Read says
Read reads up to len(b) bytes from the File. It returns the number of bytes read and an error, if any. EOF is signaled by a zero count with err set to io.EOF.

I don't see this behaviour when using InMemFile to fake a filesystem; InMemFile returns the EOF in the last read of nonzero number of bytes before EOF.

Here's a short program, afero.go

package main

import (
    "fmt"
    "io/ioutil"
    "os"
    "path"

    "github.com/spf13/afero"
)

type file interface {
    Read([]byte) (int, error)
}

func read(f file) {
    b := make([]byte, 4096)
    n, err := f.Read(b)
    fmt.Printf("n:%v, err:%v\n", n, err)
    n, err = f.Read(b)
    fmt.Printf("n:%v, err:%v\n", n, err)
}

func main() {
    f := afero.MemFileCreate("foo")
    f.WriteString("a\nb\nc\nd\n")
    f.Seek(0, 0)
    read(f)

    workdir, err := ioutil.TempDir("", "eof_test")
    if err != nil {
        fmt.Printf("couldn't open tempdir")
    }
    defer os.RemoveAll(workdir)
    g, err := os.Create(path.Join(workdir, "foo"))
    g.WriteString("a\nb\nc\nd\n")
    g.Seek(0, 0)

    read(g)
}

Output of the program is:

go run afero.go
n:8, err:EOF
n:0, err:EOF
n:8, err:<nil>
n:0, err:EOF

You can see in the second pair of lines the normal EOF behaviour is to return the number of bytes, and nil; then in the second read, no bytes read and the EOF error.

InMemFile returns EOF in both calls.

[RFC] Adding S3 support

Hello,

Do you have any plan to support S3 bucket ? or other remote FS ?

extend CopyOnWriteFs to be able to use other backends as an overlay

Currently CopyOnWriteFs can only use MemMapFs as a backend on the overlay.

It would be great to be able to use other backends. Specifically it's very useful to have two directories on disk layered with one of them being writable. This is easily accomplished using BasePathFs and OsFs but we need support for it in CoWFs.

Integrate AppVeyor

Please integrate AppVeyor for Windows tests. I already tried it with my account but of course can't integrate it with this repo (multiple tests are failing, also see here #26). I added a working config file to the repo so the setup for you should be easy.

TestWalk fails on Windows

I know you just committed fixes for the Windows errors and AppVeyor appears happy, but I'm seeing errors on TestWalk:

$ go test -v github.com/spf13/afero
...
--- FAIL: TestWalk (0.00s)
        path_test.go:67: Walk outputs not equal!
        path_test.go:69: MemMapFS
        path_test.go:70: /tmp/afero afero 0 true <nil>
                \tmp\afero\test.txt test.txt 446 false <nil>

        path_test.go:69: OsFs
        path_test.go:70: /tmp/afero afero 0 true <nil>
                \tmp\afero\renamefrom renamefrom 4 false <nil>
                \tmp\afero\test.txt test.txt 446 false <nil>

I'm on Windows 7 with go1.5.1 windows/amd64. I've run go get -u -v github.com/spf13/afero to make sure all dependencies are up-to-date.

It looks like you should be using filepath.Join and friends.

File interface is missing Sync() method

Fs interface needs to support context.Context for all API calls

Proper implementations of Afero for rpc backed filesystems will require passing a context.Context.

See: https://godoc.org/golang.org/x/net/context

The `afero.TempDir` defaults to `os.TempDir` if the `dir` argument is empty

In most scenarios this will work 100% fine. But problem comes in once we work with Remote file systems like currently experimental SFTP.

MemMapFs.OpenFile should handle O_CREATE flag

OpenFile just calls Open to virtually open a file read-only with no parsing of the flags.
At a minimum it should at least see if O_CREATE is one of the flags passed and create the file instead.

failing to build

The build started to break at commit: 5aa5058

Output from my go build

$ go build
# github.com/spf13/afero
./memmap.go:51: unknown mem.File field 'name' in struct literal
./memmap.go:51: unknown mem.File field 'memDir' in struct literal
./memmap.go:51: unknown mem.File field 'dir' in struct literal
./memmap.go:69: undefined: mem.MemFileCreate
./memmap.go:88: impossible type assertion:
    *File does not implement File (missing Close method)
./memmap.go:120: impossible type assertion:
    *File does not implement File (missing Close method)
./memmap.go:128: undefined: DirMap
./memmap.go:147: invalid pointer type *File for composite literal
./memmap.go:171: invalid pointer type *File for composite literal
./memmap.go:202: impossible type assertion:
    *File does not implement File (missing Close method)
./memmap.go:202: too many errors

I'm guessing the re-organization messed up a few things?

Version: go version go1.5.1 linux/amd64

Hugo build is broken on Windows

Looks to be Afero-related, and it happened recently.

See gohugoio/hugo#1680

Bug in Mem file implementation for ReadDir/ReadDirNames?

The following code

    fs := &afero.MemMapFs{}
    if err := fs.MkdirAll("/a/b", 0777); err != nil {
        log.Fatalf("unable to create dir:%s", err)
    }
    log.Println(fs)

    infos, err := afero.ReadDir("/", fs)
    if err != nil {
        log.Fatalf("Error with Readdir:%s", err)
    }
    log.Println(infos)

prints a nil slice for contents of / rathern than a fileinfo for /a. The MkdirAll seems to work

    2015/03/23 09:09:13 &{map[/:0xc20805a120 /a/b:0xc20805a060 /a:0xc20805a0c0] 0xc20801e400}
    2015/03/23 09:09:13 []

Any specific reason a Walk function is not implemented within the package?

I can see that Fs interface doesn't specify LStat which is used in the implementation of Walk in the standard library so any Walk implementation on afero would not be 100% compatible with the standard library

However, Walk is a very common requirement for filesystems and so I wonder if there's a specific reason afero chooses not to provide one. If no, would it be ok for me to add an implementation & file a pull request?

sftp_test_go

I assume this is failing ?
I am needing to connect to a sftp server, that does not use ssh key pair, and thought i woudl give this a try, but the fact that the test for SFTP is stubbed out makes me think its stubbed out because it is not ready.

The test is trying to setup a local SFTP server. I am working against this SFTP server.
http://docs.ipswitch.com/WS_FTP_Server71/ReleaseNotes/index.htm?k_id=ipswitch_com_ftp_documents_worldwide_ws_ftpserverv71releasenotes

Can you let me know please.

"go test -v ./..." fails on Windows

While investigating gohugoio/hugo#1669 (comment), I first tried using Readdirnames(0) in hugo/commands/new_test.go to see what actually got created, and after seeing it returns [content] on Linux, but returns [C:\Users\Veronica\AppData\Local\Temp\blog\content] on Windows, I tried to use afero.Walk() to see what files/directories actually gets created in TestDoNewSite_error_force_dir_inside_exists(), I ran into a runtime error of "invalid memory address or nil pointer deferenced" on Windows 8.1 (amd64) running go1.5.1.

I then decided to try go test -v ./... on afero itself, and got the following results:

=== RUN   TestRead0
--- PASS: TestRead0 (0.00s)
=== RUN   TestOpenFile
--- PASS: TestOpenFile (0.00s)
=== RUN   TestMemFileRead
--- PASS: TestMemFileRead (0.00s)
=== RUN   TestRename
--- FAIL: TestRename (0.00s)
    fs_test.go:160: File was not renamed to renameto
=== RUN   TestRemove
--- FAIL: TestRemove (0.00s)
    fs_test.go:178: OsFs: Remove() failed: remove /tmp/afero/test.txt: The process cannot access the file because it is being used by another process.
=== RUN   TestTruncate
--- PASS: TestTruncate (0.00s)
=== RUN   TestSeek
--- PASS: TestSeek (0.00s)
=== RUN   TestReadAt
--- PASS: TestReadAt (0.00s)
=== RUN   TestWriteAt
--- PASS: TestWriteAt (0.00s)
=== RUN   TestReaddirnames
--- FAIL: TestReaddirnames (0.00s)
    fs_test.go:325: remove /tmp/afero\test.txt: The process cannot access the file because it is being used by another process.
=== RUN   TestReaddirSimple
--- FAIL: TestReaddirSimple (0.00s)
    fs_test.go:389: file does not exist
=== RUN   TestReaddir
--- FAIL: TestReaddir (0.00s)
    fs_test.go:421: open /tmp/afero/we/have/to/go/deeper: The system cannot find the path specified.
=== RUN   TestWalk
--- FAIL: TestWalk (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0x49c8ea]

goroutine 17 [running]:
testing.tRunner.func1(0xc082060a20)
    c:/go/src/testing/testing.go:450 +0x178
github.com/spf13/afero.TestWalk.func1(0x78f440, 0xa, 0x0, 0x0, 0x2130028, 0xc0820021d0, 0x0, 0x0)
    C:/Users/Veronica/GoPath/src/github.com/spf13/afero/fs_test.go:466 +0x6a
github.com/spf13/afero.Walk(0x78f440, 0xa, 0xc082037f18, 0x2134790, 0xc082002e30, 0x0, 0x0)
    C:/Users/Veronica/GoPath/src/github.com/spf13/afero/fs.go:200 +0xc9
github.com/spf13/afero.TestWalk(0xc082060a20)
    C:/Users/Veronica/GoPath/src/github.com/spf13/afero/fs_test.go:472 +0x1c9
testing.tRunner(0xc082060a20, 0x92a840)
    c:/go/src/testing/testing.go:456 +0x9f
created by testing.RunTests
    c:/go/src/testing/testing.go:561 +0x874

goroutine 1 [chan receive]:
testing.RunTests(0x81c158, 0x92a720, 0xf, 0xf, 0x2134800)
    c:/go/src/testing/testing.go:562 +0x8b4
testing.(*M).Run(0xc08206ded8, 0x0)
    c:/go/src/testing/testing.go:494 +0x77
main.main()
    github.com/spf13/afero/_test/_testmain.go:82 +0x11d
exit status 2
FAIL    github.com/spf13/afero  0.321s