splitbrain / watcher

Watcher is a daemon that watches specified files/folders for changes and fires commands in response to those changes. It is similar to incron, however, configuration uses a simpler to read ini file instead of a plain text file. Unlike incron it can also recursively monitor directories. It's also written in Python, making it easier to hack.

About Watcher

Watcher is a daemon that watches specified files/folders for changes and fires commands in response to those changes. It is similar to incron, however, configuration uses a simpler to read ini file instead of a plain text file. Unlike incron it can also recursively monitor directories.

It's written in Python, making it easier to hack.

Requirements

You need Python 2.7 and the pyinotify library.

In Ubuntu (and Debian) you can install these with:

sudo apt-get install python python-pyinotify

Configuration

See the provided watcher.ini file for an example job configuration. The config file should reside in /etc/watcher.ini or ~/.watcher.ini. You can also specify the path to the config file as a command line parameter using the --config option.

If you edit the ini file you must restart the daemon for it to reload the configuration.

Starting the Daemon

Make sure watcher.py is marked as executable:

chmod +x watcher.py

Start the daemon with:

./watcher.py start

Stop it with:

./watcher.py stop

Restart it with:

./watcher.py restart

If you don't want the daemon to fork to the background, start it with

./watcher.py debug

watcher's Issues

Errno=No space left on device (ENOSPC)

I'm watching /var/www/html folder which contains 2 laravel projects.
[Watcher brings me hundreds of lines similar to this:

2017-10-19 08:54:02,971 pyinotify ERROR] add_watch: cannot watch /var/www/html/hu.myproject/database WD=-1, Errno=No space left on device (ENOSPC)

df -h output

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       32G  5.0G   25G  17% /
devtmpfs        884M     0  884M   0% /dev
tmpfs           903M     0  903M   0% /dev/shm
tmpfs           903M  8.3M  895M   1% /run
tmpfs           903M     0  903M   0% /sys/fs/cgroup
tmpfs           181M     0  181M   0% /run/user/1000
tmpfs           181M     0  181M   0% /run/user/1001

Any ideas or somebody encountered this already?

Too many to handle

Hey,

First: Great and helpful tool. Using this for one year and it was working most of the time.

Setting:
Debian
FTPd Server
Customer uploads pictures
watcher triggers a console application which makes a SQL Insert into our MySQL
a daemon handles this files and moves them where i want the files

Problem:
New customer upload sometime more than 200 Files at once
watcher seems not to be able to handle it
sometimes around 80 files don't get paste into MySQL

So i guess because of too many actions watcher can't queue the actions anymore

Suggestions? I will now check again what could be the problem.

Thanks!

never ending loop

Hi
Great product, but it seems to be missing a simple method of excluding certain files from triggering the action/command.

For example, I am using Watcher to trigger a simple script that launches Grive. Grive is a FOSS google drive alternative for linux. It saves it state in two files, .grive and .grive_state, which are also located in the google drive folder I am watching, Grive doesn't include any options to override the location of these files. These are updated every time Grive uploads/deletes/downloads etc, this then triggers watcher, which triggers Grive and round and round we go.

I have added entries to the watcher.ini file, but is just ignores them.
excluded=/home/kevin/googledrive/.grive,/home/kevin/googledrive/.grive_state

Please can you extend the excluded option to include files and types of files.

create and modify

Hello,
I noticed that :

if I create a new file in the monitored folder the watcher send me two event
create
modify

Why?

Python3 Support added

Hi, i've adapted the fork by DeepSpace2/Watcher, now watcher.py works with modern Python3 distributions:

https://github.com/klemens-u/Watcher-Python3

Doesn't work with Dropbox

Hi, i'm trying to watch some files recursively in my dropbox folder but that doesn't run my command it's like the script doesn't see the created file.

If i change my directory it works so i thing it's specific to Dropbox.

If you know what's the problem tell me and i'll code it myself

Thanks

Segmentation fault

I did # echo "123132" >> /data/zip/1.txt

# watcher.py debug
2016-05-31 17:55:04.736179 Daemon started
2016-05-31 17:55:04.736284 zip: /data/zip
Segmentation fault

my conf

[DEFAULT]
logfile=/tmp/watcher.log
pidfile=/var/run/watcher.pid

[zip]
watch=/data/zip
events=all
recursive=true
autoadd=true
excluded=
command=ls -l $filename

Log

# more /tmp/watcher.log
2016-05-31 17:42:21.123884 Daemon started
2016-05-31 17:42:21.123970 zip: /data/zip
Traceback (most recent call last):
  File "/usr/bin/watcher.py", line 384, in <module>
    daemon.start()
  File "/usr/bin/watcher.py", line 124, in start
    self.run()
  File "/usr/bin/watcher.py", line 273, in run
    excluded  = self.config.get(section,'excluded')
  File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'excluded' in section: 'zip'
2016-05-31 17:42:22.195822 Daemon started
2016-05-31 17:42:22.195937 zip: /data/zip
Traceback (most recent call last):
  File "/usr/bin/watcher.py", line 384, in <module>
    daemon.start()
  File "/usr/bin/watcher.py", line 124, in start
    self.run()
  File "/usr/bin/watcher.py", line 273, in run
    excluded  = self.config.get(section,'excluded')
  File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'excluded' in section: 'zip'
2016-05-31 17:42:24.283982 Daemon started
2016-05-31 17:42:24.284062 zip: /data/zip
Traceback (most recent call last):
  File "/usr/bin/watcher.py", line 384, in <module>
    daemon.start()
  File "/usr/bin/watcher.py", line 124, in start
    self.run()
  File "/usr/bin/watcher.py", line 273, in run
    excluded  = self.config.get(section,'excluded')
  File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'excluded' in section: 'zip'
2016-05-31 17:42:27.365618 Daemon started
2016-05-31 17:42:27.365717 zip: /data/zip
Traceback (most recent call last):
  File "/usr/bin/watcher.py", line 384, in <module>
    daemon.start()          
  File "/usr/bin/watcher.py", line 124, in start
    self.run()
  File "/usr/bin/watcher.py", line 273, in run
    excluded  = self.config.get(section,'excluded')
  File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'excluded' in section: 'zip'
2016-05-31 17:43:52.965675 Daemon started
2016-05-31 17:43:52.965796 zip: /data/zip
Traceback (most recent call last):
  File "/usr/bin/watcher.py", line 384, in <module>
    daemon.start()
  File "/usr/bin/watcher.py", line 124, in start
    self.run()
  File "/usr/bin/watcher.py", line 273, in run
    excluded  = self.config.get(section,'excluded')
  File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'excluded' in section: 'zip'
2016-05-31 17:46:47.676795 Daemon started
2016-05-31 17:46:47.676903 zip: /data/zip
2016-05-31 17:48:59.228131 Daemon started
2016-05-31 17:48:59.228233 zip: /data/zip
2016-05-31 17:54:41.934230 Daemon started
2016-05-31 17:54:41.934344 zip: /data/zip

Watcher tries to move watched directory itself...

Maybe just a dumb misconfiguration by a semi-noob (me ;-) )

I use Watcher to move photos from my Dropbox folder into my Darktable import folder as they come in
(emulating iCloud photostream as suggested in this thread: https://discussions.apple.com/thread/3729503?start=15&tstart=0)

Relevant lines from watcher config:
watch=/home/me/Dropbox/Camera Uploads
events=create,delete,modify,open
recursive=true
autoadd=true
command=mv $filename /home/me/Pictures/Darktable_Import

The files are moved as intended, but Watcher seems to try to move the watched dir itself...
Log is full of messages like:

mv: cannot move '/home/thewanderer/Dropbox/Camera Uploads' to '/home/thewanderer/Pictures/Darktable_Import/Camera Uploads': Directory not empty

(the dir is not empty because Watcher already tried to move the whole dir into the destination folder once before I stopped it.)

Also a lot of

mv: cannot stat '/home/me/Dropbox/Camera Uploads/2016-10-27 15.27.36 HDR.jpg': No such file or directory

Should the mv command look different, or is Watcher being quirky? ;-)

Infinite loop when invoke "ls -la" in watched directory

Not sure if it's a bug or not, but just in case let me provide details:

1. Get watcher (I downloaded it 09/05/2015)

cd /tmp; git clone https://github.com/splitbrain/Watcher

1. Create below .ini file:

sudo vim /etc/watcher.ini

[DEFAULT]
logfile=/tmp/watcher.log
pidfile=/tmp/watcher.pid
[job1]
; directory or file to watch.  Probably should be abs path.
watch=/tmp/test
events=close
excluded=
recursive=true
autoadd=true
command=ls -l $filename

1. Create watched dir:

mkdir /tmp/test

1. Start the daemon and tail life the logs:

cd /tmp/Watcher; ./watcher.py start
tail -F /tmp/watcher.log

1. Go to the watcher directory and issue a "ls -la" command:

cd /tmp/test; ls -la

Expected behaviour The log shows nothing, since no file was closed (maybe I'm wrong in this assumption...)
Found behaviour The daemon starts writing the log without stopping:

Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
Close nowrite:  /tmp/test
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0
total 0

What happens if...

Hey Andreas,

Awesome trick you've done here. May I ask you what would happen in the following situation?

I wish to do a 2-way-file sync between two NFS servers. Now for this I could use only your watcher, but I would like to use something that is a bit more sophisticated than triggering scp commands. I was thinking about unison. Easy to use and easy to set up, and still does a pretty good job. However I want to trigger it with watcher and on file create events.

Now I want an upload folder to be watched and clone the files to the other backup server when they are uploaded. What happens with the following event line?

• User starts uploading a 2 GB file
• My webapplication starts receiving the file and creates it in the uploads folder
• A file is created but obviously with not all the data yet.

My question is: Is watcher going to get triggered by this broken file as well or it is intelligent enough to wait until the upload is finished? If not, then....

• User uploads the file
• The webapp, when the upload is ready, can copy the file to a different folder
• Watcher can watch that different folder and when the file is copied then it can trigger unison

What do you think? Is the first method working or am I forced to use the 2nd option?

Thanks in advance!

Bert

Loglevel ?

Hey Dev. Team!

I was wondering, is there any option to set a log level? I wish to see in the logs all the events of what watcher does. I'm chowning files, and I want to track'em in the log. Currently, it shows only errors.

Regards:
Bert

Spaces in path

When a space character is present in the path or file name then the watcher will return an error.

After creating document named "Untitled Document 2" in /tmp following log was created

command: cannot access ‘/tmp/Untitled’: No such file or directory
command: cannot access ‘Document’: No such file or directory
command: cannot access ‘2’: No such file or directory

regexp in exclude?

Seems like regular expressions for excluding files is not working right.
I mean regexps like those mentioned here:
https://github.com/seb-m/pyinotify/blob/master/python2/examples/exclude.py

Are they tested?

For example I am excluding in jobs.yml:
['/home/foo/bar/','^/home/foo/bar/temp.']
or even:
['/home/foo/bar/','/home/foo/bar/temp.']

but I'm still getting notified for excluded files matching the regexp.
(I'm getting notifications for files like '/home/foo/bar/temp-1-2-3').

Folder/Files that changes names fast

Hi,

I am using sftp for a couple of windows computer and watcher might have a problem. Its not fast enough :) When creating a new file or folder:

chmod: cannot access '/data/files/yo/Neuer Ordner/Neues Textdokument.txt': No such file or directory
chmod: cannot access '/data/files/yo/Neuer Ordner/Neues Textdokument.txt': No such file or directory
chmod: cannot access '/data//files/yo/Neuer Ordner': No such file or directory

It seems to me that the folders get that fast renamed that watcher don't know them after they changed the name.

watcher configuration:

events=create,close,modify
command=chmod g+rw,o-rwx $filename

Getting Errno=No space left on device in the logs (ENOSPC)

When I start the script, it lists a Errno=No space left on device message for every single directory that is under the top watch folder I specified. Once I get the Daemon started message (which takes a while due to all the errors it lists for each folder) things seem to work just fine. Any idea how I can resolve this issue?

There is plenty of space on the drive so it is not a space issue for sure

File "/usr/lib64/python2.7/ConfigParser.py", line 370, in getboolean

System

• CentOS 7.1
• Package python-2.7.5-58.el7.x86_64 installed
• Package python-inotify-0.9.4-4.el7.noarch installed

What did I miss during installation?

2017-10-17 11:09:31.756856 Daemon started
2017-10-17 11:09:31.757472 web:nginx: /var/www/html
Traceback (most recent call last):
  File "/bin/watcher", line 384, in <module>
    daemon.start()
  File "/bin/watcher", line 124, in start
    self.run()
  File "/bin/watcher", line 271, in run
    recursive = self.config.getboolean(section,'recursive')
  File "/usr/lib64/python2.7/ConfigParser.py", line 370, in getboolean
    raise ValueError, 'Not a boolean: %s' % v
ValueError: Not a boolean:

Renaming folder

Hello,

I think there is a problem with watcher since when renaming a folder it still keeps in mind its old name :

./watcher.py debug
Daemon started
job1: /home/user/test
Creating: /home/user/test/folder
Moved from: /home/user/test/folder
Moved to: /home/user/test/renamed_folder
Creating: /home/user/test/folder/file.txt

Stopped working and i don't know where to start debugging

Hi,

I use this to watch a directory for ftp file uploads and performing a wget to a HTTP API notifying of new upload.
This was working just fine but something broke and i can't find it.
When I tail my xferlog i see uploads but the watcher.log remains emtpy except for the lines

2017-07-17 14:49:48.526305 Daemon started
2017-07-17 14:49:48.526542 job1: /mnt/datastore/fswatch/

However if i start using /watcher.py debug it does work and i see file creations on screen.
So in debug mode it works, in daemon mode it does not. Any ideas?

events "write_close" and "close"

Hello, my web hoster has updated from centos 6 to 7.
I survey one folder
upload file to one temp folder
When file is all update (and only ...) I move to another folder and load another script

I dont understand why watcher dont wait upload is complete to begin move file in new folder ^^
my event is
events=write_close,delete,close

(I try severals things , but nothing good ;-( )

Whats events i must use ?

Can someone help me ?
Thanks

events of job1 trigger action of job 2

Hi,

I wrote a config file with two jobs for the same directory :

watcher.ini
[job_write]
watch=/some/dir
events=create,modify,move_to,write_close
recursive=true
autoadd=true
excluded=
command=/some/path/in_create.sh $filename $nflags $tflags

[job_delete]
watch=/some/dir
events=delete,move_from
recursive=true
autoadd=true
excluded=
command=/some/path/in_delete.sh $filename $nflags $tflags

The scripts called write logs and copy file to another server :

in_create.sh
#!/bin/bash
LOGFILE=/var/log/watcher.log
echo "$(date +%Y-%m-%d);$(date +%H:%M:%S);$1;$3;$2;in_create.sh" >> $LOGFILE
(...)

in_delete.sh
#!/bin/bash
LOGFILE=/var/log/watcher.log;
echo "$(date +%Y-%m-%d);$(date +%H:%M:%S);$1;$3;$2;in_delete.sh" >> $LOGFILE
(...)

I expect, when I create a directory or file, that only in_create.sh will be called, but the logs show that both scripts in_create and in_delete are called.

I found a workaround by joining the 2 jobs in one and making a script who checks the nature of events and redirects to the right script, but I guess this behaviour is not normal.

I see in my logs lines like this one :
$mkdir /some/dir/test

2016-03-11;13:16:18;/some/dir/test;IN_CREATE|IN_ISDIR;1073742080;in_create.sh
2016-03-11;13:16:18;/some/dir/test;IN_CREATE|IN_ISDIR;1073742080;in_delete.sh

The second line should not be there, should it ?