Expire daemons when open file descriptors count reach a treshold about gradle

[hmijail]

In building the Hyperledger Besu project we found that, in the latest MacBooks Pro, some unit tests fail because the open file descriptors go over 10240 (as reported by lsof). This happens both with macOS 10.15 and 10.14.6, and with JDKs 11 and 12. Gradle 5.4.1.

The same build works well in 1-2 year old MacBooks - in there I see the open file descriptors going up to "only" about 8500.

For the time being we're using _JAVA_OPTIONS=-XX:-MaxFDLimit to allow the tests to finish even with those high numbers of descriptors. We're still not sure why the file descriptors go so high nor why they go even higher in the newer Macs, but if the tests didn't fail we wouldn't even notice.

[hmijail]

After a bit more testing, I see that the open FDs piling in the gradle daemons are thousands of pipes, shown by lsof like this:

...
java 99069 mija 4292u KQUEUE count=0, state=0x8
java 99069 mija 4293 PIPE 0xcefa84f33b5d7d39 16384 ->0xcefa84f33b5d7c79
java 99069 mija 4294 PIPE 0xcefa84f33b5d7c79 16384 ->0xcefa84f33b5d7d39
java 99069 mija 4295u KQUEUE count=0, state=0x8
java 99069 mija 4296 PIPE 0xcefa84f33b5d84b9 16384 ->0xcefa84f33b5d8579
java 99069 mija 4297 PIPE 0xcefa84f33b5d8579 16384 ->0xcefa84f33b5d84b9
...

I assume that the daemon is being sent its tasks (running unit tests in this case) through those pipes, and then somehow they don't get closed.

Is there anything that we can do to make the daemon close them?

[big-guy]

Which Java process are you looking at? Is it the Gradle daemon (the long lived process, stays alive between builds) or the Test Worker daemon (runs unit tests) or something else?

In the past, the failures we've seen have been around custom tasks leaking open file descriptors and not Gradle itself. Does the number of open file descriptors go down once the build is complete?

One difference between old Macs and new ones could be the number of available processors. On the builds that fail, you could try limiting the number of parallel tasks (e.g., --max-workers=2). It could be that more tasks/tests are running in parallel on newer machines and they're opening lots of file descriptors.

[hmijail]

Which Java process are you looking at? Is it the Gradle daemon (the long lived process, stays alive between builds) or the Test Worker daemon (runs unit tests) or something else?

Judging by the way the PIDs appeared and disappeared, I would say this was the Test Worker daemon.
I have a script which continually dumps the lsof of all the processes named java. If interested I can generate the sampled history of PIDs and FDs and send it to you.

In the past, the failures we've seen have been around custom tasks leaking open file descriptors and not Gradle itself. Does the number of open file descriptors go down once the build is complete?

In general it seems to mostly grow, though rarely it does ratchet back a bit.
Looks like the daemons are stopped and restarted when different gradle subprojects are run, which generally keeps the FDs from accumulating too much.

One difference between old Macs and new ones could be the number of available processors. On the builds that fail, you could try limiting the number of parallel tasks (e.g., --max-workers=2). It could be that more tasks/tests are running in parallel on newer machines and they're opening lots of file descriptors.

My machine is "old" so it doesn't cause the failure, though I do see the FDs growing to >8500. One of the problematic machines did behave even worse with --no-daemon.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.

[stale]

This issue has been automatically closed due to inactivity. If you can reproduce this on a recent version of Gradle or if you have a good use case for this feature, please feel free to reopen the issue with steps to reproduce, a quick explanation of your use case or a high-quality pull request.