Comments (6)
In building the Hyperledger Besu project we found that, in the latest MacBooks Pro, some unit tests fail because the open file descriptors go over 10240 (as reported by lsof
). This happens both with macOS 10.15 and 10.14.6, and with JDKs 11 and 12. Gradle 5.4.1.
The same build works well in 1-2 year old MacBooks - in there I see the open file descriptors going up to "only" about 8500.
For the time being we're using _JAVA_OPTIONS=-XX:-MaxFDLimit to allow the tests to finish even with those high numbers of descriptors. We're still not sure why the file descriptors go so high nor why they go even higher in the newer Macs, but if the tests didn't fail we wouldn't even notice.
from gradle.
After a bit more testing, I see that the open FDs piling in the gradle daemons are thousands of pipes, shown by lsof
like this:
...
java 99069 mija 4292u KQUEUE count=0, state=0x8
java 99069 mija 4293 PIPE 0xcefa84f33b5d7d39 16384 ->0xcefa84f33b5d7c79
java 99069 mija 4294 PIPE 0xcefa84f33b5d7c79 16384 ->0xcefa84f33b5d7d39
java 99069 mija 4295u KQUEUE count=0, state=0x8
java 99069 mija 4296 PIPE 0xcefa84f33b5d84b9 16384 ->0xcefa84f33b5d8579
java 99069 mija 4297 PIPE 0xcefa84f33b5d8579 16384 ->0xcefa84f33b5d84b9
...
I assume that the daemon is being sent its tasks (running unit tests in this case) through those pipes, and then somehow they don't get closed.
Is there anything that we can do to make the daemon close them?
from gradle.
Which Java process are you looking at? Is it the Gradle daemon (the long lived process, stays alive between builds) or the Test Worker daemon (runs unit tests) or something else?
In the past, the failures we've seen have been around custom tasks leaking open file descriptors and not Gradle itself. Does the number of open file descriptors go down once the build is complete?
One difference between old Macs and new ones could be the number of available processors. On the builds that fail, you could try limiting the number of parallel tasks (e.g., --max-workers=2
). It could be that more tasks/tests are running in parallel on newer machines and they're opening lots of file descriptors.
from gradle.
Which Java process are you looking at? Is it the Gradle daemon (the long lived process, stays alive between builds) or the Test Worker daemon (runs unit tests) or something else?
Judging by the way the PIDs appeared and disappeared, I would say this was the Test Worker daemon.
I have a script which continually dumps the lsof
of all the processes named java
. If interested I can generate the sampled history of PIDs and FDs and send it to you.
In the past, the failures we've seen have been around custom tasks leaking open file descriptors and not Gradle itself. Does the number of open file descriptors go down once the build is complete?
In general it seems to mostly grow, though rarely it does ratchet back a bit.
Looks like the daemons are stopped and restarted when different gradle subprojects are run, which generally keeps the FDs from accumulating too much.
One difference between old Macs and new ones could be the number of available processors. On the builds that fail, you could try limiting the number of parallel tasks (e.g.,
--max-workers=2
). It could be that more tasks/tests are running in parallel on newer machines and they're opening lots of file descriptors.
My machine is "old" so it doesn't cause the failure, though I do see the FDs growing to >8500. One of the problematic machines did behave even worse with --no-daemon.
from gradle.
This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.
from gradle.
This issue has been automatically closed due to inactivity. If you can reproduce this on a recent version of Gradle or if you have a good use case for this feature, please feel free to reopen the issue with steps to reproduce, a quick explanation of your use case or a high-quality pull request.
from gradle.
Related Issues (20)
- Secondary variants cannot be created lazily HOT 3
- Misleading/confusing doc on `withType` eager/lazy API overloads HOT 1
- Improve Javadoc of ConfigurableFilePermissions.html#unix(int) HOT 3
- No included builds contain this plugin HOT 1
- Document that JUnitOptions and JUnitPlatformOptions become abstract with Gradle 9
- Document that BaseForkOptions becomes abstract with Gradle 9
- `DefaultDevelocityBuildLifecycleService` shouldn't force IP compatibility for build HOT 2
- Custom converter/adapter for @Option HOT 1
- Gradle should not validate isolated projects when isolated projects is disabled HOT 3
- Tests unrunnable when in directory with characters not encodable by system encoding HOT 6
- Where do you download gradle-wrapper.jar? HOT 1
- Provide a flag to disable parallel configuration for IP mode in favor of configure-on-demand HOT 2
- Deadlock in parallel tasks configuration in IP mode
- De-incubate Streamed Values in TAPI requests
- Java compilation errors are missing from build scans failure page HOT 1
- Parallel CC store should be an opt-in
- "Invalid Java installation found" while auto-provisioning a JDK HOT 2
- Unable to compile Scala 3 macro code - `NoClassDefFoundError` for `javax.sql.DataSource` HOT 2
- Provide a real module names for generated type-safe accessor HOT 1
- '/usr/lib/jvm/java-17-openjdk-amd64' does not provide the required capabilities: [JAVA_COMPILER] HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gradle.