Alpha Windows support about watchman

[wez]

OK folks, thanks for following along; I'm going to promote us from alpha to beta on Windows, so I'm closing this issue.

The install section of the docs has more information about downloading and installing the beta, as well as encouraging folks to file issues if/when they encounter them.

https://facebook.github.io/watchman/docs/install.html#download-for-windows-beta

[kumarharsh]

@wez: I'm writing a plugin for vscode: graphql-for-vscode which relies on watchman. Users have been using it for about a month now, and till date there's not been a single issue. Pretty good work by your team 👍

[igorsvee]

Can confirm that hot reloading works on Win 8.1 + genymotion + react-native 0.34.1 using https://ci.appveyor.com/api/buildjobs/kravw77nw7fqhvio/artifacts/watchman.zip. Thanks guys.

[wez]

This link works correctly:

https://ci.appveyor.com/api/projects/wez/watchman/artifacts/watchman.zip?branch=master&job=Environment:+WATCHMAN_WIN7_COMPAT%3D&pr=false

The issue is that the default for pr is true so it's trying to download artifacts from the not-yet-ready cmake changes I'm hacking on in another pr :-/

[wez]

@sunnylqm it's not clear how that symlink related issue has anything to do with alpha windows support. Please open a separate issue so that we can keep this one focused on the status of the windows port of Watchman, thanks!

[muloka]

@wez Alternatives to watchman that are suitable for Windows

Anyway since writing my comment I've dabbled in both Powershell and syncthing

Using the two I've managed to arrive at a solution that works.

[lednhatkhanh]

Hi, where can I find the latest windows version?

[wez]

@masaeedu would you mind opening a separate issue to ask those questions? This one is already pretty overloaded and long

[wez]

Windows is not currently supported. We have no immediate plans to add support for it. We would welcome a contribution to add this support if someone has the time and inclination to produce it.

[wez]

To support Windows, we first need to tidy up our handling of case-insensitive filesystems; that work is in-progress, but we still have no immediate plans for Windows at this time.

[wez]

We're tracking the list of things that would need to be tackled for Windows support in this wiki page:
https://github.com/facebook/watchman/wiki/Changes-required-for-Windows-support

[pclouds]

This is something you guys may want to watch out when using ReadDirectoryChangesW [1]

I dropped ReadDirectoryChangesW because maintaining a 'live' file system cache became more and more
complicated. For example, according to MSDN docs, ReadDirectoryChangesW may report short DOS 8.3
names (i.e. "PROGRA~1" instead of "Program Files"), so a correct and fast cache implementation would
have to be indexed by long and short names...

[1] http://article.gmane.org/gmane.comp.version-control.git/216195

[ziriax]

You might want to consider using the NTFS Change Journal. It is rock solid. I used it for Evolver, a closed source version control system that I developed for a game company in 2002.

It seems a Python wrapper is already made

[wez]

cc: @dsp

I just pushed https://github.com/wez/watchman/tree/win with my hacking in progress; it's been sitting on my laptop for a while and I just rebased on master as of today and make sure that it still builds on both osx and windows.

You will need the microsoft visual C++ compiler to build it; you can use the community edition for this.
The compiler tools (cl, nmake and so on) must be in your path.
To run the integration tests you will need arcanist and php installed and in your path.

To build and test:

cd watchman
nmake -f winbuild\Makefile check
arc test

The integration tests are a bit rough and ready largely because of the filesystem semantics on windows are not POSIX; there are a lot of errors around attempting to delete dirs while handles are still open on them. There are also some errors with / instead of \ in path names. These are for the most part cosmetic.

The current status is 26 of 74 tests failed.

If you're interested in helping with hacking and testing, you are most welcome! We're at the cleaning things up stage.

If you're interested in making a client connect to it, you can connect to a named pipe for the current user; for example, my watchman instance listens at \\.\pipe\watchman-wez. The code to select the default name of the pipe can be found here: https://github.com/wez/watchman/blob/win/main.c#L456

Once connected, you use the standard JSON or BSER watchman protocol to communicate with the server.

[ziriax]

Great! I just tried to compile and run watchman.exe, but the code seems to use Windows 8 specific APIs, while we're still at Windows 7. For example, the API function GetOverlappedResultEx is used. Is this by design?

[wez]

Windows 8 has a number of API calls that make it easier to port posix style applications and we're currently using those. In addition, I didn't have easy access to Windows 7 while I was prototyping this. I'm not opposed to making changes to have this run on Windows 7.

[ziriax]

It seems the only Win8 specific API function call is the GetOverlappedResultEx. At first sight it seems you can just replace this with the old GetOverlappedResult function, since your timeout is either 0 or INFINITE, so the timeout maps to the BOOL bWait argument of the old function.

I was able to compile and run the watchman fine on my Win7 machine, and the log shows that watching indeed works.

I was not yet able to get the hgwatchman Mercurial extension running on Windows, but I suspect I have a conflict between 32-bit and 64-bit version. I'll dig deeper.

[wez]

@ziriax great! I just (force) pushed some fixes to my branch to pick up the python build and test bits. You'll also see how to establish the named pipe connection you'll need for hgwatchman--you have to use os.open because the higher level file abstraction in python doesn't work properly with named pipes

[wez]

I remembered that I added make.bat in my branch as a short cut, so you can just run:

make

to have it run nmake with the right makefile for you. This script will also source the visual studio build environment. We're moving towards having the tests use python, so it is best if you have python in your path when you build; I'm using python 2.7.

[wez]

Current status is 18 of 73 failing, and a quick scan of the output looks like they're all / vs \ path related in the test expectations, or because we don't yet have pcre in the build and aren't smart enough to skip the pcre tests if support is missing.

I fixed a couple of race conditions and issues with relative roots and window filesystem convention handling.

I'll take a look at the GetOverlappedResult thing as well a bit later tonight.

[wez]

OK, the most recent run is down to just 6 failing tests.
I've also pushed a commit that tries to do the right thing on Windows 7; it's not as simple as just calling GetOverlappedResult as that will impact the ability of the service to handle concurrent clients (more details can be found in that commit)

[stefanpenner]

@wez if I recall some modes of windows FS watching (at-least the flags that are used in libuv) trigger change events on a file when attaching a hard link to it.

If possible, could watchman on windows not share this same behavior?

[stefanpenner]

If memory serves (it was quite some time ago), attaching a hard link mutates an attribute on the file: https://github.com/libuv/libuv/blob/188e0e94ac8374a5e177e1407052ed1b26c528a6/src/win/fs-event.c#L49

[stefanpenner]

It's also worth noting, my above statement might just be describing a symptom of an entirely different problem. So feel free to disregard as you feel appropriate.

[wez]

[FYI: am currently traveling and may not be super responsive]

As of the most recent commit on wez/watchman:win, watchman is buildable on windows and basically functions (as in, I have only performed the most basic test as shown below) with hgwatchman.

To get hgwatchman to work, you'll need to copy the python/pywatchman dir from the windows watchman tree to hgwatchman/hgwatchman/pywatchman and then apply this patch to hgwatchman:

diff --git a/hgwatchman/__init__.py b/hgwatchman/__init__.py
--- a/hgwatchman/__init__.py
+++ b/hgwatchman/__init__.py
@@ -170,6 +170,8 @@
     if normalize:
         foldmap = dict((normcase(k), k) for k in results)

+    hg_slash = '.hg' + os.sep
+
     for entry in result['files']:
         fname = entry['name']
         if normalize:
@@ -183,7 +185,7 @@
         fmode = entry['mode']
         fexists = entry['exists']
         kind = getkind(fmode)
-        if fname == '.hg' or fname.startswith('.hg/'):
+        if fname == '.hg' or fname.startswith(hg_slash):
             continue

         if fname not in results:

Then run this in the root of the hgwatchman repo:

python setup.py build_py -c -d .  build_ext -i

Then:

hg --config extensions.hgwatchman=z:\fb\hgwatchman\hgwatchman --config watchman.mode=on st

[wez]

@stefanpenner https://github.com/wez/watchman/blob/win/watcher/win32.c#L142 is what we're using for our win32 watcher.

We're really at the mercy of what windows reports here. The nature of these things is that neither the kernel nor watchman itself can reliably maintain enough state to guarantee that we can quash that kind of a notification without risking making the wrong choice and not notifying you of a similar but otherwise notify-able change.

Our philosophy on these situations in watchman is that we are conservative and err on the side of over notifying rather than missing a notification.

[stefanpenner]

Our philosophy on these situations in watchman is that we are conservative and err on the side of over notifying rather than missing a notification.

this sounds very reasonable :) If we run into issues, i will explore it in more details and report back. Thank you for the link.

[ziriax]

For a similar project I used the NTFS change journal. As far as I recall, this is very reliable. But since it logs all changes to a partition, we used to have a dedicated partition for our source files, and another one for the generated files.

[wez]

Pull request for hgwatchman to support windows:
https://bitbucket.org/facebook/hgwatchman/pull-requests/3

[wez]

@ziriax yeah, we've often talked about how it would be great if the filesystems themselves gave us an API for these kinds of operations; it would help to eliminate some of the TOCTOU issues that we have to deal with in the watchman service.

I haven't looked at the NTFS change journal API, but watchman does have a reasonably modular way to plug in an alternative watcher implementation (watcher/win32.c is currently the only windows implementation); it might be interesting to try building a change journal watcher to contrast/compare. It does sound a bit more painful to configure and deploy, so it probably would not be the default mode of operation. In addition, the ReadDirectoryChangeW API should work with more filesystems.

I don't think we're actively going to try the change journal ourselves, but I'm definitely open to reviewing and evaluating that contribution :-)

[ziriax]

Well, that's the nice thing about the NTFS change journal, since the file system writes entries in the journal, you do not need to "watch" anything. You just ask the file system about what changes happened between two "timestamps". No software has to run to constantly monitor the changes.

We found the directory notification system to be rather unreliable. But we used Windows XP in those days, so a lot could be improved by now.

-----Original Message-----
From: "Wez Furlong" [email protected]
Sent: ‎30/‎07/‎2015 19:22
To: "facebook/watchman" [email protected]
Cc: "Peter Verswyvelen" [email protected]
Subject: Re: [watchman] Windows support? (#19)

@ziriax yeah, we've often talked about how it would be great if the filesystems themselves gave us an API for these kinds of operations; it would help to eliminate some of the TOCTOU issues that we have to deal with in the watchman service.
I haven't looked at the NTFS change journal API, but watchman does have a reasonably modular way to plug in an alternative watcher implementation (watcher/win32.c is currently the only windows implementation); it might be interesting to try building a change journal watcher to contrast/compare. It does sound a bit more painful to configure and deploy, so it probably would not be the default mode of operation. In addition, the ReadDirectoryChangeW API should work with more filesystems.
I don't think we're actively going to try the change journal ourselves, but I'm definitely open to reviewing and evaluating that contribution :-)
—
Reply to this email directly or view it on GitHub.

[jorangreef]

I looked into the NTFS change journal for a similar project, but as far as I tried it needs Administrator privileges to work? There doesn't seem to be a way to get at the raw data without it.

[ziriax]

@jorangreef Yes, typically I would write a Windows service running with the correct rights for accessing and controlling the change journal. The service itself could be queried by any user. The service itself should be installed using admin rights, could be part of the installer...

[jorangreef]

Thanks @ziriax

[wez]

I merged the windows branch into master. We still need to fixup the trigger tests, but it appears functional for those able to build it. I'll see if we can get a more rigorous build process hooked up for this to make it easier for folks to test.

[sschuberth]

I'll see if we can get a more rigorous build process hooked up for this to make it easier for folks to test.

Maybe integrate with AppVeyor for the Windows build / tests in addition to Travis CI?

[wez]

@sschuberth thanks; checking it out :)

[stefanpenner]

We use it for ember-cli, it does the trick. Not quite as stable as travis, but \o/.

[wez]

I'll need to do some work to make the complete test suite run on there, but here's a link to the first .exe not produced by my dev vm:

https://ci.appveyor.com/api/buildjobs/huojrdlyqqghnb6g/artifacts/watchman.exe

It does not have PCRE support, but it does have the enhanced wildmatch support that we introduced in 3.7.

If you're using node, we haven't looked at adjusting the nodejs client yet, but it should be pretty trivial, anyone want to try that?

https://nodejs.org/api/net.html#net_server_listen_path_callback implies that the nodejs net/socket layer knows how to talk to named pipes if you pass in the correct path. If that doesn't work, a simple file handle opened on the pipe path that watchman get-sockname returns should be usable instead.

[wez]

100% of tests passed in this windows CI run!
https://ci.appveyor.com/project/wez/watchman/build/1.0.9

[sschuberth]

Yay, congrats!

[felixrieseberg]

Congrats!

[fabioparodi]

Hello
watchman works on my system (win7 64 bits).
Using the hgwatchman extension, 'hg status' runs 10x faster than before.
The graphical refresh of TortoiseHG does not benefit at all - still very slow

[wez]

@fabioparodi great to hear! Can you share some details on how many files and directories are in the repo that you're using? Is this on NTFS or FAT32?

[barnyrebal]

hi, any guide how to install watchman on install? thanks

[feibinyang]

https://facebook.github.io/watchman/docs/install.html
@barnyrebal this?

[barnyrebal]

deadlock - the link on git project watchman point to here, and here you point back to install on git :)

[Grisou13]

@barnyrebal
Goto this link http://bit.ly/watchmanwinalpha (from first comment in this thread....)
Unzip the content to somewhere that is refernced in your path (e.g C:\Program files ...i will let you check the internet for more info)
Then your setup ! :)

[ZebraFlesh]

So is this still alpha support? Sounds like it works, (but hard for me to test since facebookarchive/nuclide#259 blocks me).

[wez]

It wouldn't be a stretch to call it beta, but we're calling it alpha on the grounds that we don't have a lot of direct experience using it for anything real on Windows (Linux and Mac are our main direct user population).

There are a couple of open issues that we need to run down, so it's definitely not quite up to the same level as the other supported systems.

We definitely need and value feedback, positive or negative

[timofischer81]

Before installing watchman on windows 10 64bit, crawling file system took more than 50s. So this is improvement but unfortunately still not very fast.

React packager ready.

[14:46:28] Crawling File System (24154ms)
[14:46:28] Building in-memory fs for JavaScript
[14:46:28] Building in-memory fs for JavaScript (164ms)
[14:46:28] Building in-memory fs for Assets
[14:46:28] Building in-memory fs for Assets (111ms)
[14:46:28] Building Haste Map
[14:46:28] Building (deprecated) Asset Map
[14:46:28] Building (deprecated) Asset Map (36ms)
[14:46:28] Building Haste Map (112ms)
[14:46:28] Building Dependency Graph (24553ms)
[14:46:45] request:/index.android.bundle?platform=android&dev=true&hot=false
[14:46:45] find dependencies
[14:46:45] find dependencies (618ms)
[14:46:45] transform
transforming [========================================] 100% 527/527
[14:46:46] transform (749ms)
[14:46:46] request:/index.android.bundle?platform=android&dev=true&hot=false (1495ms)

[wez]

@timofischer81 please file a separate issue and include your watchman log files in that issue, along with a count of the number of files and directories contained in the project that you're watching

[victorpavlenko]

Watchman: Watchman was not found in PATH. See https://facebook.github.io/watchman/docs/install.html

Windonws 10/ SSD

[Ultarius]

@Xurma2 Had the same issue. I also had a node package called watchman globally installed. When I removed it, it was solved. Could be the same issue.

[johniak]

@Ultarius @Xurma2 The watchman fron npm global repository is not the same as facbook watchman

[sunnylqm]

@wez My mistake! This is the right link facebook/react-native#7022
I think it's windows port related. If not, I'll open a new one. Thank you!

[wez]

@sunnylqm I'd like someone to open a separate issue against the watchman project for that; it's not appropriate to track here in this one (as it is rather overloaded and has a lot of subscribers), and I can't track it over in the RN issue(s). Just to set expectations, we don't have anyone actively working to support Watchman on Windows today, so we'll need some really good logs and diagnostics to come up with an idea of what may be going wrong.

The only suggestion I have in the meantime is trying a very recent build of watchman to pick up 30f295e which fixes some divergence if another process has recently had a file open when watchman tries to look at it.

[sunnylqm]

@wez OK. I'll try to contact the original reporter, or transfer that post to a new issue here myself. And how can I try recent build? Clone the source and build myself?... Thank you!

[wez]

Hmm, I thought the bitly link in the main description above was auto-updating to be current, I'm not sure if that is the case.

This is link is the most recent windows build from our CI system: https://ci.appveyor.com/api/buildjobs/kravw77nw7fqhvio/artifacts/watchman.zip

[upsuper]

Hmm, I thought the bitly link in the main description above was auto-updating to be current, I'm not sure if that is the case.

It doesn't seem to me that's the case.

• http://bit.ly/watchmanwinalpha redirects to
• https://ci.appveyor.com/api/buildjobs/bd0fppc1sh70emkq/artifacts/watchman.zip redirect to
• https://appveyorcidata.blob.core.windows.net/wez-25370/watchman/b11/bd0fppc1sh70emkq/watchman.zip?sv=2014-02-14&sr=c&sig=91vkxVcvwaWe05R2AmTni1NRu3%2F1xvhPcd0j%2BvdBR8k%3D&st=2016-09-23T11%3A32%3A13Z&se=2016-09-23T11%3A38%3A13Z&sp=r

The Last-Modified field of the last url shows Sat, 08 Aug 2015 19:22:02 GMT.

Instead, the link in your comment shows Tue, 23 Aug 2016 17:20:41 GMT for Last-Modified.

[waynebloss]

Are you guys officially supporting Windows now or not?

The first comment still reads "Windows is not currently supported" and this issue is still open, so I am assuming the answer is No?

Is this project even necessary for Windows, since Windows had a file-change notifications API way before the Mac OS or Linux had them?

Thanks.

[wez]

@waynebloss at the moment the folks working on Watchman don't run it on Windows for any real purpose. We're anticipating that this will change in the future, which is why we've added support and have CI configured and so on. Until that changes and we've seen what really happens to the system outside of our integration test suite, we feel that it would be disingenuous to label it more than alpha or perhaps beta.

Watchman provides both a real-time file change subscription mechanism (echoing what the native OS facilities provide, but in a cross-platform way) and functionality to query the view of the filesystem without talking to the filesystem. The latter part of this may sound strange, but when you are dealing with a very large directory tree, the act of looking at a "cold" portion of it can block for a non-trivial amount of time.

Whether watchman is necessary depends on your needs.

[muloka]

@wez out of curiosity, got any recommendations of alternative tools for Windows?

[wez]

@muloka do you mean alternatives to watchman, or just tools for windows?

[kumarharsh]

@lednhatkhanh The link is in the original post above: http://bit.ly/watchmanwinalpha

[wez]

Figured out that we can use this link to always reference the latest build:

https://ci.appveyor.com/api/projects/wez/watchman/artifacts/watchman.zip?branch=master&job=Environment:+WATCHMAN_WIN7_COMPAT%3D

(updated the description to reflect this)

[masaeedu]

@wez The clockspec on Windows looks a bit weird; it doesn't seem to match the two segment clockspec shown in the docs. Is there some special meaning for each

I get something with four segments when I run watchman since D:\depot\git\agentautodeditor\sampledata n:foo '':

{
    "version": "4.9.0",
    "is_fresh_instance": false,
    "clock": "c:1496775649:11508:1:63",
    "files": []
}

Additionally, this clockspec seems to increment every time I run that command, whereas from my understanding of the docs logical clock IDs should not increment unless files have actually changed in the directory. I've been using watchman clock D:\depot\git\agentautodeditor\sampledata instead, which seems to maintain a stable clock when I haven't changed anything.

Could you point me to some detailed docs on the anatomy of a clock id, and whether this is how things should be working?

[chidionuekwusi]

The link is broken

[AdrianoCahete]

The link is broken

This is working: https://ci.appveyor.com/api/buildjobs/41p1lupk97crwvm0/artifacts/watchman.zip