crash during w_root_perform_age_out on OS X about watchman

[wez]

can you also collect the watchman log(s) from this system, and ideally the git hash or at least the watchman version number so we can make sure we're looking at the right code?

[wez]

(edited the description to highlight the pertinent part of the crash logs)

[dturner-tw]

Version is 2.9.9 -- here are some logs (but I don't know which ones precede the crashes; once we start getting those, we kill watchman and restart). The git sha this corresponds to should be d924624.

1411390858: tid=2024162064 file limit is 2048 kern.maxfilesperproc=10240
1411390858: tid=2024162064 raised file limit to 10240
1411390858: tid=2024162064 failed to parse json from /var/folders/tf/wfsrwyg94_52s4xcgp1qb4nc0000gp/T//.watchman.arass.state: unable to open /var/folders/tf/wfsrwyg94_52s4xcgp1qb4nc0000gp/T//.watchman.arass.state: No such file or directory
1411390858: tid=48873472 path /Users/arass/workspaces/birdcage is on filesystem type hfs
1411393538: tid=48873472 sync_to_now: /Users/arass/workspaces/birdcage/.git/.watchman-cookie-tw-mbp-arass-499-1-4823 timedwait failed: Operation timed out
1411398571: tid=46772224 sync_to_now: /Users/arass/workspaces/birdcage/.git/.watchman-cookie-tw-mbp-arass-499-1-52846 timedwait failed: Operation timed out
1411400009: tid=2024162064 file limit is 2048 kern.maxfilesperproc=10240
1411400009: tid=2024162064 raised file limit to 10240
1411400009: tid=2024162064 path /Users/arass/workspaces/birdcage is on filesystem type hfs
1411486646: tid=2024162064 file limit is 2048 kern.maxfilesperproc=10240
1411486646: tid=2024162064 raised file limit to 10240
1411486646: tid=2024162064 path /Users/arass/workspaces/birdcage is on filesystem type hfs
1411489759: tid=274206720 path /Users/arass/workspaces/science is on filesystem type hfs
1411489894: tid=274206720 sync_to_now: /Users/arass/workspaces/birdcage/.git/.watchman-cookie-tw-mbp-arass-34459-1-4790 timedwait failed: Operation timed out
1411499501: tid=2024162064 file limit is 2048 kern.maxfilesperproc=10240
1411499501: tid=2024162064 raised file limit to 10240
1411499501: tid=2024162064 path /Users/arass/workspaces/science is on filesystem type hfs
1411499501: tid=2024162064 path /Users/arass/workspaces/birdcage is on filesystem type hfs

[wez]

sync_to_now: /Users/arass/workspaces/birdcage/.git/.watchman-cookie-tw-mbp-arass-499-1-4823 timedwait failed: Operation timed out

I wouldn't expect this to happen unless the client is specifying a very low timeout for the query, or if the .git directory is somehow not being watched. In the past, this was easy to do accidentally by listing .git in the ignore_dirs configuration, but this should not longer be possible. Is there a .watchmanconfig for this repo too?

Regardless, the crash is the thing I'd like to track in this particular issue (unless this happens to be something like a deadlock between the sync and the age out, but I don't see indications of that in the gist you posted)

[dturner-tw]

The timeout is 600 ms -- is that too low? There is no .watchmanconfig.

[wez]

We've got a handle on the query timeouts portion of this; it's a class of error around fsevents going bad, either because there are too many clients, or because of a change in security context. #60 is probably the closest thing we have to talk to that, although it is focused on large number of watches. The thread there links to a couple of diffs in review on our public phabricator instance that will help avoid and better surface this class of error.

I haven't seen any other incidences of this class of crash. Once the patches I mentioned above land, it would be great if you could get those running and see if this issue is still lurking, and we can debug from there.

[dturner-tw]

Sorry for the delay; I was on vacation. I'll upgrade us internally and let you know if we still have issues.

[dturner-tw]

We upgraded to b5bc4ab but are still seeing crashes:
https://gist.github.com/dturner-tw/c66e246d6448693aecbc

[dturner-tw]

Another user just got the same crash as https://gist.github.com/dturner-tw/c66e246d6448693aecbc#file-watchman_2014-12-20-152606_localhost-crash

[wez]

Finally(!) found some cycles to dig in to this and pushed a fix.
Thanks for your patience!

[dturner-tw]

Thanks!