Coder Social home page Coder Social logo

fsnotify's Introduction

fsnotify is a Go library to provide cross-platform filesystem notifications on Windows, Linux, macOS, BSD, and illumos.

Go 1.17 or newer is required; the full documentation is at https://pkg.go.dev/github.com/fsnotify/fsnotify


Platform support:

Backend OS Status
inotify Linux Supported
kqueue BSD, macOS Supported
ReadDirectoryChangesW Windows Supported
FEN illumos Supported
fanotify Linux 5.9+ Not yet
AHAFS AIX aix branch; experimental due to lack of maintainer and test environment
FSEvents macOS Needs support in x/sys/unix
USN Journals Windows Needs support in x/sys/windows
Polling All Not yet

Linux and illumos should include Android and Solaris, but these are currently untested.

Usage

A basic example:

package main

import (
    "log"

    "github.com/fsnotify/fsnotify"
)

func main() {
    // Create new watcher.
    watcher, err := fsnotify.NewWatcher()
    if err != nil {
        log.Fatal(err)
    }
    defer watcher.Close()

    // Start listening for events.
    go func() {
        for {
            select {
            case event, ok := <-watcher.Events:
                if !ok {
                    return
                }
                log.Println("event:", event)
                if event.Has(fsnotify.Write) {
                    log.Println("modified file:", event.Name)
                }
            case err, ok := <-watcher.Errors:
                if !ok {
                    return
                }
                log.Println("error:", err)
            }
        }
    }()

    // Add a path.
    err = watcher.Add("/tmp")
    if err != nil {
        log.Fatal(err)
    }

    // Block main goroutine forever.
    <-make(chan struct{})
}

Some more examples can be found in cmd/fsnotify, which can be run with:

% go run ./cmd/fsnotify

Further detailed documentation can be found in godoc: https://pkg.go.dev/github.com/fsnotify/fsnotify

FAQ

Will a file still be watched when it's moved to another directory?

No, not unless you are watching the location it was moved to.

Are subdirectories watched?

No, you must add watches for any directory you want to watch (a recursive watcher is on the roadmap: #18).

Do I have to watch the Error and Event channels in a goroutine?

Yes. You can read both channels in the same goroutine using select (you don't need a separate goroutine for both channels; see the example).

Why don't notifications work with NFS, SMB, FUSE, /proc, or /sys?

fsnotify requires support from underlying OS to work. The current NFS and SMB protocols does not provide network level support for file notifications, and neither do the /proc and /sys virtual filesystems.

This could be fixed with a polling watcher (#9), but it's not yet implemented.

Why do I get many Chmod events?

Some programs may generate a lot of attribute changes; for example Spotlight on macOS, anti-virus programs, backup applications, and some others are known to do this. As a rule, it's typically best to ignore Chmod events. They're often not useful, and tend to cause problems.

Spotlight indexing on macOS can result in multiple events (see #15). A temporary workaround is to add your folder(s) to the Spotlight Privacy settings until we have a native FSEvents implementation (see #11).

Watching a file doesn't work well

Watching individual files (rather than directories) is generally not recommended as many programs (especially editors) update files atomically: it will write to a temporary file which is then moved to to destination, overwriting the original (or some variant thereof). The watcher on the original file is now lost, as that no longer exists.

The upshot of this is that a power failure or crash won't leave a half-written file.

Watch the parent directory and use Event.Name to filter out files you're not interested in. There is an example of this in cmd/fsnotify/file.go.

Platform-specific notes

Linux

When a file is removed a REMOVE event won't be emitted until all file descriptors are closed; it will emit a CHMOD instead:

fp := os.Open("file")
os.Remove("file")        // CHMOD
fp.Close()               // REMOVE

This is the event that inotify sends, so not much can be changed about this.

The fs.inotify.max_user_watches sysctl variable specifies the upper limit for the number of watches per user, and fs.inotify.max_user_instances specifies the maximum number of inotify instances per user. Every Watcher you create is an "instance", and every path you add is a "watch".

These are also exposed in /proc as /proc/sys/fs/inotify/max_user_watches and /proc/sys/fs/inotify/max_user_instances

To increase them you can use sysctl or write the value to proc file:

# The default values on Linux 5.18
sysctl fs.inotify.max_user_watches=124983
sysctl fs.inotify.max_user_instances=128

To make the changes persist on reboot edit /etc/sysctl.conf or /usr/lib/sysctl.d/50-default.conf (details differ per Linux distro; check your distro's documentation):

fs.inotify.max_user_watches=124983
fs.inotify.max_user_instances=128

Reaching the limit will result in a "no space left on device" or "too many open files" error.

kqueue (macOS, all BSD systems)

kqueue requires opening a file descriptor for every file that's being watched; so if you're watching a directory with five files then that's six file descriptors. You will run in to your system's "max open files" limit faster on these platforms.

The sysctl variables kern.maxfiles and kern.maxfilesperproc can be used to control the maximum number of open files.

fsnotify's People

Contributors

abustany avatar arp242 avatar bep avatar code0x58 avatar cpuguy83 avatar davecheney avatar fsouza avatar horahoradev avatar howeyc avatar jolheiser avatar maranix avatar mattn avatar nathany avatar nitrocao avatar nshalman avatar oozie avatar paulhammond avatar pieterd avatar pratik32 avatar r-darwish avatar rchiossi avatar rfratto avatar robfig avatar saschagrunert avatar shogo82148 avatar tklauser avatar tmc avatar vmirage avatar zeldovich avatar zhsso avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fsnotify's Issues

cannot watch non-existent files

Maybe I'm mis-understanding the semantics of fsnotify, but I'm surprised by the fact that I can't watch non-existent files. E.g., I can't watch a filename and then get a CREATE event when it's created. Or, if I watch a file and it's replaced via a REMOVE then I can no longer track it. This means that to reliably track changes to a file I must watch the file's directory!?

Example:

package main

import (
        "time"

        "gopkg.in/fsnotify.v1"
)

func main() {
        w, _ := fsnotify.NewWatcher()
        w.Add("foo")
        go func() {
                for ev := range w.Events {
                        println("Event: ", ev.String())
                }
        }()
        go func() {
                for err := range w.Errors {
                        println("Error: ", err)
                }
        }()
        time.Sleep(60 * time.Second)

}

Test:

$ go version
go version go1.3 linux/amd64
$ rm foo
$ go run notify.go &
[1] 6990
$ echo hello >foo
# I was expecting a CREATE event at this point...
$ kill %1
[1]   Exit 2                  go run notify.go

$ go run notify.go &
[2] 7004
$ echo hello >foo
Event:  "foo": WRITE
Event:  "foo": WRITE
# This time I got the event because foo already existed, but why do I get two write events?
$ rm foo
$ Event:  "foo": REMOVE
$ echo hello >foo
# I was expecting a CREATE event at this point...
$ kill %2

Improved "rename" support

As mentioned here: howeyc/fsnotify#104

Currently, it's not straightforward to detect a rename correctly. In most cases, one will see this:

RENAME file1
CREATE file2

Yet this doesn't guarantee that file1 and file2 are the same file (there could be a file creation in between, or file1 could be moved outside a watched folder and file2 could be moved in). To work around this, one could store the os.FileInfo for every watched file and use os.SameFile to compare it with the new file. This is a bit cumbersome though, as it requires keeping a lot of os.FileInfo structs in memory + bookkeeping them.

However, some platforms provide a way to atomically see what was the old name and what's the new name of a file. To quote @nathany:

On Linux there is a "cookie" used to tie the RENAME FROM and TO events together, but this isn't currently exposed by fsnotify. I still need to do more research across each platform.

This is a tracker issue to see if there is sufficient cross-platform support to enable this in the default API.

Better handling of file-close events on Linux

Requested by @spf13 and @extemporalgenome.

@spf13: on linux at least, all you need to do the proper tracking of vim/any-other-kind-of-editor "done writing" tracking is watch for IN_CLOSE_WRITE and IN_MOVE_TO on the directory

@extemporalgenome if it's a lazy editor, that just updates in place, IN_CLOSE_WRITE catches it; if it's a "write to temporary and atomically rename" editor, IN_MOVE_TO catches it

The trouble is in doing this in a cross-platform way. And some users may still want IN_MODIFY (eg. to show the progress of a file writing).

consistency with stdlib io api

Hi,

In the current api, the NewWatcher() is spawning a new goroutine when initialized that yields its results through a channel. This is slightly inconsistent with other io packages in the standard library. Tcp sockets, Udp sockets, etc are all blocking (e.g. on Read()) and leave the goroutines up to the developer of the library. Furthermore, the error handling in its current form is also slightly inconsistent with the rest of the standard library. The Go standard library should try to stay consistent where possible.

What if the Watcher instead included a watcher.Watch() method that returned a event and err tuple, just like in many other packages? That way, the developer could choose himself/herself if wrapping it in a goroutine is required, thus possibly reducing the overhead where not strictly required.

E.g.:
func (w *Watcher) Watch() (Event, error)

Just a thought.

remove Event.String()?

The specifics of how this string is formatted doesn't seem like it belongs in the library.

Replace String() with a helper function in the example and test suite.

OpCode 6

When the watching directory is deleted, sometimes I get OpCode 6, and not 4. I don't see it as a constant, what is that op code?

Polling fallback

Whether or not fsnotify implements polling support itself, some thought should be given into how polling could work as an alternative to native OS events.

GoConvey uses polling exclusively to avoid the "too many files" error #8.

"we walk the file system every quarter second and use the sum of the last mod time stamp and size of each go file as a quick comparison. Along the way we make note of new and deleted packages, all the while skipping 'ignored' packages. It's actually quite speedy." - @mdwhatcott, Gopher Slack

If you use an approach like what I've done in GoConvey, make sure to count empty directories as well... - @mdwhatcott

https://github.com/smartystreets/goconvey/blob/master/web/server/watcher/scanner.go

@pifantastic reports that node's gaze (https://github.com/shama/gaze#errors) library detects EMFILE errors and falls back to polling.

Polling could be opt-in for network file systems (NFS), Vagrant or Plan 9 -- where OS events don't work or are unavailable.

Deadlock due to unbuffered channel

Mac OS 10.10.1, go version go1.4 darwin/amd64, pulling from tip which as of this writing is go-fsnotify/fsnotify@f582d92

This program will deadlock, even though I believe it's a correct use of the API:

package main

import (
    "fmt"

    "github.com/go-fsnotify/fsnotify"
)

func main() {
    watcher, err := fsnotify.NewWatcher()

    if err != nil {
        panic(err)
    }
    defer watcher.Close()

    watcher.Add("./")

Loop:
    for {
        select {
        case event := <-watcher.Events:
            fmt.Println(event)
            break Loop
        case err := <-watcher.Errors:
            fmt.Println(err)
            break Loop
        }
    }
}

(Save a file in the current directory to trigger an event and see the deadlock.)

It is fixed by giving the internal done channel a buffer of 1 (which seems to be the case in inotify.go).

Ease of use

"Some of you may be familiar with existing file monitoring solutions... they are a pain in the butt to use, they are complex, they are not user friendly, and they have a lot of race conditions that you have to be aware of as a developer."

"We've extensively tested it, we rolled it out for several months to all our developers... what's actually on disk, what does Watchman think is on disk... when we found discrepancies we fixed it."

Scaling Source Control at Facebook by Durham Goode

kqueue improvements

What have I learned from experimenting with kqueue directly (primarily on OS X)? #23

  • fsnotify doesn't watch for Extend, Link or Revoke events (tracked in #519 now).

  • fsnotify has a 100ms timeout on reads, but instead we may be able to just block until kqueue has events available (nil timespec). this could save a little CPU time when idle, see: howeyc/fsnotify#58

    • closing the kq (on OS X) results in the read failing with Kevent: interrupted system call, so blocking in a goroutine shouldn't delay shutdown.
  • fsnotify could add multiple file descriptors at once (such as all files in a directory), which may be a little more efficient then adding each one individually.

  • investigated Udata which C programs sometimes use to point to the watched filename, but it's probably not worthwhile from Go (and NetBSD defines Udata differently than the other BSDs).

Other improvements are specific to fsnotify, such as less mutexes #13 and bug fixes #14.

FSEvents on macOS

Requested by @robfig at howeyc/fsnotify#54 in order to watch large trees.

To be implemented as a stand-alone repository for experimentation (#23) before determining how to incorporate it into fsnotify.

System requirements:

  • OS X 10.6 or better (matching the system requirements of Go 1.3). Though some features require 10.7, or perhaps even higher.
  • Likely require cgo with Xcode installed.

Prior work:

Reference:

Kqueue refactoring regression

I've spent the entire day trying to reproduce the error reported in https://github.com/go-fsnotify/fsnotify/pull/43 in a consistent fashion. But I can't...

I've tried with a tiny revel app (just revel new) - no issue. I've tried with a slightly larger (booking) - no issue. But with our large-ish revel app I still have the issue.

I tried the following with an instrumented revel watcher[1]:

  1. Launch the app. revel run ourapp
  2. Go to /robots.txt and wait for revel to rebuild and serve the file.
  3. Without touching any files of the project, refresh the browser (f5) and see what happens.

With the kqueue branch (71b4293) I got this output the first time I tried:
http://paste2.org/MaDA2xsn
and this the second time:
http://paste2.org/Ih9zspp4

So it behaves differently in two different runs even though I'm doing the exact same thing. ๐Ÿ˜ข

Then I switched back to the master branch (ca50e73) and I consistently get the following:
http://paste2.org/Fmm6pvVn

So with the master branch there are no rebuilds triggered by changes to tmp/main.go or routes/routes.go. With master the only events are the folder changes to app/tmp and app/routes. Can this give you any clue?

[1] Where https://github.com/revel/revel/blob/develop/watcher.go#L174-L176 was changed to this:

case ev := <-watcher.Events:
    fmt.Println("ev", ev.String())
    if w.rebuildRequired(ev, listener) {
        fmt.Println("rb", ev.String())
        refresh = true

os/fsnotify

Motivation

For some time there has been a desire to provide file system notifications in the standard library. See
golang/go#4068 (was https://code.google.com/p/go/issues/detail?id=4068).

This would allow the tools that ship with Go to use it:

It may start as an internal package while working to solidify the API. Once exposed in the standard library, fsnotify will need to adhere to the Go 1.0 guarantee for backwards compatibility. It is not yet ready for that kind of guarantee.

System requirements

Go 1.3 requires Linux 2.6.23 or OS X 10.6.

  • If we use IN_NONBLOCK (#5) fsnotify would require Linux 2.6.27.
  • FSEvents supports 10.6, but watching individual files requires 10.7. (kFSEventStreamCreateFlagFileEvents)

CLA

All contributors to fsnotify are asked to sign Google's CLA. @bradfitz added many contributors to Go AUTHORS in the past (see howeyc/fsnotify#85).

New AUTHORS since then will need to be added.

fsnotify.v1 error on get / build

When trying to get v1 on Windows 7 the following error appears: (golang 1.2 amd64) upgrading to golang 1.3 fixed the issue

go\src\gopkg.in\fsnotify.v1\windows.go:420: undefined: syscall.ERROR_MORE_DATA

Additional filters

I would like to implement filters like these in a separate utility package:

  • shell pattern matches, such as *.go
  • excluding hidden files and directories (.git, .hg)
  • throwing out duplicate events on a file (occurring within a second)

(My attempt to incorporate them into fsnotify itself is here: howeyc/fsnotify#65)

fsnotify.Write is not catching updates on Mac

if event.Op&fsnotify.Write == fsnotify.Write {
    log.Println("modified file:", event.Name)
}

Events trigged: RENAME -> CREATE -> CHMOD -> CHMOD -> CHMOD

The exact example is not catching the event after file updates.

Missing create events on OS X

@elgs reported this issue at howeyc/fsnotify#82

In the a directory, when I run touch a s d f g h k l, it outputs:

2014/02/02 03:31:00 event: "/Volumes/User/Home/Desktop/a/a": CREATE
2014/02/02 03:31:00 event: "/Volumes/User/Home/Desktop/a/d": CREATE
2014/02/02 03:31:00 event: "/Volumes/User/Home/Desktop/a/s": CREATE
2014/02/02 03:31:00 event: "/Volumes/User/Home/Desktop/a/l": CREATE

kqueue doesn't have a NOTE_CREATE event so fsnotify_bsd.go is doing a ReadDir and scanning for created files. However, it someone gets confused as to which files already exist:

/Users/nathany/Desktop/a/a false
/Users/nathany/Desktop/a/s false
2014/02/01 22:26:28 event: "/Users/nathany/Desktop/a/a": CREATE!
2014/02/01 22:26:28 event: "/Users/nathany/Desktop/a/s": CREATE!
/Users/nathany/Desktop/a/a true
/Users/nathany/Desktop/a/d true
/Users/nathany/Desktop/a/f true
/Users/nathany/Desktop/a/g true
/Users/nathany/Desktop/a/h true
/Users/nathany/Desktop/a/j true
/Users/nathany/Desktop/a/k true
/Users/nathany/Desktop/a/l false
/Users/nathany/Desktop/a/s true
2014/02/01 22:26:28 event: "/Users/nathany/Desktop/a/l": CREATE!

The problem is that sendDirectoryChangeEvents does a ReadDir and sends a create event for a file. Then it calls watchDirectoryFiles which does a ReadDir as well. By this time more files have been touched so ReadDir gets a different result.

watchDirectoryFiles is marking those files as existing without sending create events. And since they have been flagged as existing, those files are never sent as create events the next time sendDirectoryChangeEvents is called.

For some reason I haven't been able to reproduce the issue on FreeBSD, which uses nearly all the same code.

File Events Notifications on Solaris

@4ad would like fsnotify to support FEN on Solaris.

While each file & directory requires an event source, the limits are much higher than kqueue:

The default limit is 64k event sources per event port per process.
The default limit for the number of event sources per system is
2^31.

The default limit for the number of event ports per process is 8k.
The default limit for the number of event ports per system is 64k.

Reference

Cannot remove watch for deleted file

Once a file is deleted, you cannot clean up watches for it. For instance, with fsnotify:

func (w *Watcher) Remove(name string) error {
    name = filepath.Clean(name)
    w.mu.Lock()
    defer w.mu.Unlock()
    watch, ok := w.watches[name]
    if !ok {
        return fmt.Errorf("can't remove non-existent inotify watch for: %s", name)
    }
    success, errno := syscall.InotifyRmWatch(w.fd, watch.wd)
    if success == -1 {
        return os.NewSyscallError("inotify_rm_watch", errno)
    }
    delete(w.watches, name)
    return nil
}

The file is gone, so even though we find the watch in w.watches we return a syscall error and don't delete the entry from w.watches.

It seems that maybe we should delete from w.watches regardless, or at least if errno is EBADF. If this sounds good, I can make a PR (and try to figure out the appropriate change for other systems (at least darwin) as well.

Remove Errors channel

As per the API document, we would like to replace the Errors channel with an Err() method and close the channel on fatal errors.

The question is how to handle non-fatal errors (if any).

Spotlight indexing causes additional events on macOS

@paulhammond reported this issue at howeyc/fsnotify#62

A temporary workaround is to add your folder(s) to the Spotlight Privacy settings.

It looks like the extra update update is being caused by mds (the daemon responsible for updating the file metadata used by spotlight). See lines 6 onwards from the output of fs_usage, which shows the name of any process performing file operations:

$ sudo fs_usage | grep TestFsnotifyAttrib.testfile
11:46:35  open              _test/TestFsnotifyAttrib.testfile                                                0.000065   fsnotify.tes
11:46:35  lstat64           _test/TestFsnotifyAttrib.testfile                                                0.000010   fsnotify.tes
11:46:35  open              _test/TestFsnotifyAttrib.testfile                                                0.000009   fsnotify.tes
11:46:35  chmod             _test/TestFsnotifyAttrib.testfile                                                0.000024   fsnotify.tes
11:46:35  access            /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000008   dbfseventsd 
11:46:35  getattrlist       /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000192   mds         
11:46:35  lstat64           /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000018   mdworker    
11:46:35  stat64            /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000013   mdworker    
11:46:35  lstat64           /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000009   mdworker    
11:46:35  getattrlist       /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000116   mdworker    
11:46:35  open              /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000024   mdworker    
11:46:35  getattrlist       /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000041   mdworker    
11:46:35  getattrlist       /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000116   mdworker    
11:46:35  getattrlist       go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile/.DS_Store    0.000009   mdworker    
11:46:35  getattrlist       go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile/.DS_Store    0.000006   mdworker    
11:46:35  getattrlist       go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile/.DS_Store    0.000005   mdworker    
11:46:35  getattrlist       go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile/.DS_Store    0.000005   mdworker    
11:46:35  getattrlist       /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000024   mds         
11:46:35  getattrlist       /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile    0.000010   mds         
11:46:35  unlink            _test/TestFsnotifyAttrib.testfile                                                0.000131   fsnotify.tes

(It's also worth noting that this test case consistently passes as of 2c4a662, because mds ignores temporary directories. But the problem is still there.)

I tried implementing throttling but found it ineffective. Spotlight will sometimes touch several files, whereas the throttling was for multiple events on the same file. At the time I tried this, NOTE_ATTRIB would trigger Write events, so it may be worth another shot (as of a100b1a).

Hopefully Spotlight will behave better with FSEvents #11.

Windows USN Journals

@pkrnjevic mentioned Windows USN Journals here: fsnotify/fsevents#1 (comment)

"On Windows NT 4.0, these tasks were accomplished with functions like FindFirstChangeNotification and ReadDirectoryChangesW. Anyone who has attempted to use these functions knows how limited they can be. The Change Journal provides a new level of detailed information for applications that need to monitor changes on an NTFS volume."

inotify blocks

readEvent() currently blocks.

See https://code.google.com/p/go/issues/detail?id=5536

@howeyc attempted to change this in 3bfa915 but reverted it 7beb451 due to reports of no new events howeyc/fsnotify#52.

This was something I added to stop readEvent from blocking forever.
Basically calling close would mean no new data came to the file descriptor
and Read would block forever.

This was an attempt to use select with a timeout so it would return after
the timeout stating whether there was new data.

User-space recursive watcher

@jbowtie requested this feature at howeyc/fsnotify#56.

Windows and FSEvents (OS X) can watch a directory tree, in fact FSEvents is built entirely for this purpose:

if you monitor a directory with FSEvents then the monitor is recursive; there is no non-recursive option - @samjacobson howeyc/fsnotify#54 (comment)

Ideally fsnotify would expose a way to use this functionality on Windows and Macs.

Unfortunately inotify (Linux, Android), kqueue (BSD, iOS), and FEN (Solaris) don't have this capability.

There is some debate as to whether or not fsnotify should include a user-space recursive watcher to make up the difference. If not, perhaps it can be implemented a separate wrapper or utility library. In any case, it's something often requested, so we should definitely be thinking about it.

Implementation

In two parts:

  1. Walk subdirectories to Add (or Remove) watches for each directory:
    • To avoid watching too much, it may be important to (optionally?) skip hidden directories (.git, .hg).
    • My proposal to configure Ops filtering globally at the Watcher level (#7) simplifies recursive watching.
  2. Listen for incoming Create events to watch additional directories as they are created (again, avoiding hidden directories).

Windows FS_MOVED_TO is CREATE action, not RENAME

Please, fix it.

This code || mask&sys_FS_MOVED_TO == sys_FS_MOVED_TO for Windows should be checked in create, not in rename.

Linux and OSX returns "create + rename" on move action. But Windows returns "rename + rename".

Event logger for research

Suggested by @matthias-stone. Log the raw events coming from the OS for a variety of actions:

  • Touching files
  • Creating directories
  • Saving files in various editors
  • Deleting a watched file/directory
  • etc.

This information can then be used to improve how fsnotify operates across platforms.

Implementation

Rather than add a verbose mode to fsnotify, this may be a separate repository with individual cmds for each adapter (cleanly implemented, getting closer to the APIs).

  • kqueue
  • Windows
  • FSEvents
  • FEN
  • inotify

Failed to Add watch to the mapped SharePoint drive

I can watch my local hard drive or mapped LAN drive, but failed to watch mapped SharePoint drive(V:). I can read/write the contents of drive V: from command line or Windows File Explore.

When I run this:

err = watcher.Add("V:")  // Windows 7 system

I got this error:

2014/09/17 16:09:43 ReadDirectoryChanges: Incorrect function.
exit status 1

The SharePoint site is like "https://my.sharepointsite.net/thename/...". Does that "HTTPS" cause this problem?

Atomic saves in text editors cause havoc, unable to watch individual files

Reported by @robfig at howeyc/fsnotify#91. See also nathany/looper#14.

When editors use atomic saves they Create a new file and then do a Rename rather than just a straight Write (like when touching a file).

Related issues:

Dealing with text editors may be out of scope of fsnotify itself, but it could at least provide recommendations in the Wiki.

@robfig also reported some other oddities, though those may be specific to kqueue on OS X.

Subtree watch on Windows

@htoothrot mentioned at howeyc/fsnotify#56 (comment)

ReadDirectoryChanges[1] has a "bWatchSubtree" argument. If this argument is true, you will receive events for all files/directories for the whole directory tree rooted at the given directory.

As watching a large/busy tree can lead to many events I thought this was worth bringing up: It may be helpful to have the buffer size configurable in some way. Right now it's a hard coded 4096 bytes. When many events fire quickly, events can be dropped when the buffer is full. There are caveats to simply increasing the size (http://stackoverflow.com/a/3250454 explains some of the issues) and it seems there's not one answer to fit all uses.

Also look into FindFirstChangeNotification

OS X: replacing a file with mv; Add returns "bad file descriptor"

Using current master, if I watch a file and then replace that file with a new one using mv new_file watched_file, I get an fsnotify.Remove event. If I respond to that event by attempting to re-add the file, I occasionally get an error: bad file descriptor. If I add a sleep or a retry it works, so I suspect this is a race condition? Even with the retry, it occasionally breaks and stops watching the changes, so there is definitely something weird happening here.

Reproduction steps (Mac OS X Mavericks):

  1. echo "wtf" > wtf
  2. go run fsnotify_bug.go wtf
  3. In another terminal: for i in seq 50; do echo $i; echo "t" > w; mv w wtf; sleep 2; done

Example program:

package main

import (
    "fmt"
    "os"
    "syscall"
    "time"

    "github.com/go-fsnotify/fsnotify"
)

func main() {
    if len(os.Args) != 2 {
        os.Stderr.WriteString("go run fsnotify_bug.go (example path)\n")
        os.Exit(1)
    }
    path := os.Args[1]

    watcher, err := fsnotify.NewWatcher()
    if err != nil {
        panic(err)
    }

    err = watcher.Add(path)
    if err != nil {
        panic(err)
    }
    for {
        select {
        case event := <-watcher.Events:
            fmt.Printf("event: %v\n", event)
            switch event.Op {
            case fsnotify.Rename, fsnotify.Remove:
                fi, err := os.Stat(event.Name)
                if err != nil {
                    fmt.Printf("stat failed; file no longer exists? %v\n", err)
                    // return
                } else {
                    s := fi.Sys().(*syscall.Stat_t)
                    fmt.Printf("stat success inode:%d size:%d\n", s.Ino, s.Size)
                }
                // UNCOMMENT: This makes it work

                for i := 0; i < 10; i++ {
                    fmt.Println("Add attempt", i)
                    err = watcher.Add(event.Name)
                    if err != nil {
                        fmt.Println("  add failed", err)
                        // return
                    } else {
                        fmt.Println("  add succeeded")
                        break
                    }
                    time.Sleep(1 * time.Millisecond)
                }
                if err != nil {
                    panic(err)
                }
            case fsnotify.Chmod, fsnotify.Write, fsnotify.Create:
                // ignore
            default:
                panic("unhandled event " + event.String())
            }

        case err := <-watcher.Errors:
            if err == nil {
                panic("unexpected nil err")
            }
            return
        }
    }
}

Removing recursive watches

I've implemented recursive watching for a few different projects. The way I've basically done this is:

  • Add a watch for the root dir
  • When there's a create event, walk the created dir and add new watches for all dirs
  • When there's a remove event, remove the watch for that dir

(I'm using fsnotify.v1.) I've realized that this isn't correct, because I don't delete the watches recursively. So I guess I need to keep track of the full tree of watches on my end as well? Maybe fsnotify could help out by exposing a 'delete all watches with this prefix' method or something.

Suggestions?

Additionally, #40 means that even if I could know all the dirs to remove watches for, I still couldn't delete them.

inotify is leaking a file descriptor

Reported by @alberts at https://codereview.appspot.com/8202043/#msg6

I noticed that TestFsnotifyDeleteWatchedDir is leaking a file descriptor:

strace -f -q -e inotify_init,open,close ./fsnotify.test
-test.cpu=1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
-test.v -test.run=TestFsnotifyDeleteWatchedDir

SIGABRT shows that all the readEvents goroutines (which are in charge of closing
the inotify fd) are still stuck in their read syscall.
This seems like a problem with the current design of Watcher. readEvents is
blocked and can't see the closed done channel.

See #5 regarding blocking behaviour.

Plan to make v0 match howeyc/fsnotify

I made the mistake of tagging v0.1.0 for a series of API changes.

By moving v0.1.0 forward to v1.0.0 and all subsequent tags, we will have:

  • gopkg.in/fsnotify.v0 will keep the same API as howeyc/fsnotify, making it easy to switch import paths in dependants. Some fixes may be backported to v0 as well.
  • gopkg.in/fsnotify.v1 the current API with some features+bugs removed.

This is just a matter of updating tags and the CHANGELOG. No code changes.

Before doing this, anyone using fsnotify.v0 should switch over to v1. @agaue

This is inspired by @davecheney's suggestion to consolidate the various repositories into one place. Inspired by @fatih, this could include opening pull requests for those importing go.exp/fsnotify or importing howeyc/fsnotify.

Sync method

There is also a question of synchronization. Ideally you want to have a "Sync()" method that guarantees that when it returns, all events for modifications made before the call to Sync have now been delivered...

I know how to build a Sync from the fsevents API. I am less sure about the others; probably it is possible but it requires some thought. It also affects the API, because you need to know when the events triggered by the Sync have stopped. If the events are coming over a channel, you need some kind of sentinel event marking the sync position. This is all unclear to me. I think fsevents gives some clear guarantees about delivery that let you build Sync.

Windows FindFirstChangeNotification seems not to give any at all. For example, you find out about file changes in a directory by watching the mtime on the files, but the docs say: "The operating system detects a change to the last write-time only when the file is written to the disk. For operating systems that use extensive caching, detection occurs only when the cache is sufficiently flushed."

That's clearly useless for interactive go command build result caching. - @rsc, https://groups.google.com/d/msg/golang-dev/bShm2sqbrTY/IR8eI20Su8EJ

Built-in event filtering of Ops

The current mechanism for filtering events is to inspect the Op bitmask:

if event.Op&fsnotify.Create == fsnotify.Create {

In howeyc/fsnotify there is a WatchFlags method to filter operations. WatchFlags was removed in 10e1440 (v0.10.0) because:

  • The implementation required a lot of extra bookkeeping while providing little benefit over filtering events as they were received.
  • As @mstum reported, it wasn't working on Windows, which was due to the bookkeeping not being implemented for files within a directory watch: howeyc/fsnotify#93 (comment).
  • There were no tests for WatchFlags.

@nightlyone pointed out that it still is desirable to specify filters so the kernel doesn't need to wakeup our thread/process for an event we'll just ignore.

Some research (API doc) reveal differences from one OS to the next:

  • kqueue doesn't have a Create Op at all (NOTE_CREATE), much less a filter for it.
  • Windows has a filter for FILE_NOTIFY_CHANGE_FILE_NAME that covers Create, Remove and Rename. An event does indicate which one of these it is, but some user-space filtering is still necessary.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.