Comments (27)
https://github.com/nunit/nunit-console/milestone/2 😄
from nunit-console.
@CharliePoole this is a problem with 3.4.1. I tried the latest master after finding that #1712 was fixed after 3.4.1 and found the same behaviour, namely: running with --agents=4
(or --agents=X
) starts 13 agent EXEs (when testing 13 assemblies), though it's true that only X of them are using the CPU at any given time. The same thing happens with and without --dispose-runners
. (None of my test classes implement IDisposable, by the way.)
What I would expect to happen is that only X agent processes exist at any given time. I don't care whether NUnit re-uses the same agent process or one process exits and a new one starts - that's an implementation detail. The issue is the amount of memory all these agent processes take, which is already slowing our build servers down a lot with 13 assemblies, but with 20 or more will become completely unmanageable. Then we would probably be forced to use --process=Separate
.
By the way, what's the difference between --process=Separate
and --process=Single
? They both seem to start only one agent process.
from nunit-console.
@shift-evgeny I promise you that it worked at one time! But that fix broke some other combinations of options and the change to fix them ended up breaking this again. I'm trying to figure out how to create a test for this so it doesn't break again. Of course, I could do an integration-style test easily, but I'd prefer to do some mocking and just have a unit test verifying that we don't try to load assemblies before we run them.
It's interesting to consider the distinction between a feature and an implementation detail. When I created --agents
I was trying to solve a problem of CPU utilization, so whether the processes existed or not was a minor detail for me. I just didn't want them all to run tests at the same time!
Considering memory, it's clear we don't even want the processes to exist simultaneously. That's what we will achieve when this is again fixed.
Someone, someday will probably want to minimize the process creation overhead and then the number of times we create and destroy a process will become a feature to implement. Not today, however, since that's a bit complicated. Each process you created would have to be tagged in terms of the .NET framework version targeted as well as it's bitness in order to decide whether to reuse it.
Regarding the other options: --process=Separate
creates a single process and agent to run all tests while --process=Single
runs them all in the NUnit process, without creating any agent. The latter option is poorly named and I added --inprocess
as a hopefully clearer synonym.
from nunit-console.
The fix was post-3.4.1 in nunit/nunit#1718, but I think it got broken again in nunit/nunit#1736 where some of the changes were undone to fix bugs that were introduced by nunit/nunit#1718.
from nunit-console.
@shift-evgeny Based on @rprouse 's comment, it looks like you should be able to use the 3.5.0-dev-03113 build until I get this fixed.
from nunit-console.
@CharliePoole where can I find that?
from nunit-console.
@shift-evgeny https://ci.appveyor.com/project/CharliePoole/nunit/build/3.5.0-dev-03113
from nunit-console.
@CharliePoole thanks, I tested that and confirmed that --agents
limits the number of agent processes, as expected! Could we get that fix in a release soon, please?
from nunit-console.
@shift-evgeny I just had to change this code based on another issue. Would you test using the latest dev build, which is currently 3.5.0-dev-03158, available on Appveyor or from MyGet.
from nunit-console.
@CharliePoole I only see builds of nunit.framework.dll and nunitlite.dll there, not nunit3-console.exe, nunit.engine.api.dll, etc. Could you confirm which assembly is supposed to contain the fix?
from nunit-console.
@shift-evgeny I guess you looked on AppVeyor for the nunit project. However, we just split the engine and console into a separate repo called nunit-console. See https://ci.appveyor.com/project/CharliePoole/nunit-console/build/3.5.0-dev-03158
For myget, see https://www.myget.org/feed/Packages/nunit
from nunit-console.
OK, tested it and it still works - the number of agents is limited to 3 with --agents=3
, as I expect. So when do you expect to publish a release that includes this fix?
from nunit-console.
(I replaced only the files in NUnit.ConsoleRunner.3.5.0-dev-03158.nupkg, so I hope that's what contained the fix. Other files are still from the earlier build.)
from nunit-console.
The 3.5 release is scheduled for the end of September. If there's a critical bug fix to release, we could include this one as well, but I don't see any issue right now that would lead to that.
from nunit-console.
@shift-evgeny Did that include nunit.engine.dll? That's what has changed.
from nunit-console.
Yes, nunit.engine.dll was in the package. So looks like it's all good. OK, we'll wait for 3.5 then.
from nunit-console.
Tested 3.5.0-ci-03195-pr-52 and then 3.5.0-dev-03179 (master d31ebef) from https://ci.appveyor.com/project/CharliePoole/nunit-console/build/artifacts.
The former did not work while the latter does 👍
Only #43 left.
from nunit-console.
FYI, only the dev suffix builds have actually made it to master. PR builds are "hypothetical" builds of master.
from nunit-console.
I see NUnit 3.5 has been released, but is there a release of NUnit Console that contains this fix? If not, when is it expected?
from nunit-console.
We are working on the nunit-console release right now and you may see it today. Note that in future, releases may not come out at the same time and version numbers will probably get out of sync. Separating releases was the main reason for splitting repositories.
from nunit-console.
I just downloaded and tested the 3.5.0 release and with 17 assemblies and --agents=3
I get 17 nunit-agent.exe processes starting up straight away, but only 3 at a time are using any CPU. They seem to exit as tests in an assembly finish running. This is an improvement on the old behaviour where all processes kept running until the end, but still, it takes extra time at start-up and extra memory. What I'd expect is that --agents=X
means only X
agent processes exist at any given time.
from nunit-console.
I can confirm this too.
Version from this https://ci.appveyor.com/project/CharliePoole/nunit-console/build/artifacts worked like a charm, while Nunit.Console 3.5 from nuget is having the old behaviour
from nunit-console.
This really needs a unit test. It seems to keep getting fixed and re-broken.
from nunit-console.
What a bummer! I'll check to see how this happened.
@shift-evgeny We had unit tests for it. But for the final proof, we need a true integration test.
from nunit-console.
Setting up a CI environment, this feature is much appreciated :)
from nunit-console.
Thanks! So what release will contain this fix and when is it expected to be released?
from nunit-console.
@shift-evgeny - This will be in 3.6, which I believe is planned for January, cc @rprouse?
from nunit-console.
Related Issues (20)
- Unlist versions 16.x to the extent possible HOT 1
- Renaming branches HOT 1
- Define console / engine runtime platform support HOT 5
- ExtensionService does not validate the content of addins files HOT 1
- Console Runner Package Tests should exercise all our extensions
- Standardize scripts across console and extension repos HOT 4
- Eliminating locking of assembly file while test is loaded HOT 37
- Eliminate .NET Core 3.1 Agent HOT 1
- Eliminate .NET 5.0 Agent HOT 1
- Add .NET 8.0 agent HOT 1
- Eliminate .NET 2.0 Agent
- Simplify MSI build process HOT 5
- Eliminate msi package HOT 1
- Develop a new CI build running under GitHub Actions HOT 12
- Unit Test Failures are not displayed prominently HOT 1
- NUnit.Console 3.17.0 and lower does not work with NUnit version 4.1.0 HOT 11
- Build using version 1.0.0-dev00001 of common NUnit cake recipe
- Unable to install NUnit.ConsoleRunner.NetCore 3.18.0 due to not found settings file 'DotnetToolSettings.xml' HOT 5
- Need target to allow re-publishing a failed symbol package HOT 1
- We have two nuget packages with the same Id: NUnit.ConsoleRunner.NetCore HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nunit-console.