Comments (16)
But having a limit even for failed test results, which needs to be disabled by using Retain long standard output/error -- the result of which is to not truncate even passing results just exacerbates the whole performance issue. It forces people to have all (failed and passing) results with full stdout/stderr just so that they can get full stdout/stderr on failed runs.
Truncating output on failed runs can mean the difference between having the data needed to fix a bug or not, so truncating it can be very debilitating so for some installations it's important to not truncate. But achieving that shouldn't mean having the performance impact of having all runs without truncated output.
The only two choices available (all runs with full output or failed runs with truncated output) are far too far apart to be workable.
As an aside, the documentation is simply wrong when it says Such output is always kept if the test failed. It's not. It's still subject to a threshold.
from junit-plugin.
This issue is a monumental facepalm π€¦ββοΈ.
So there's this underlying architectural limitation that causes the system to maybe crash due to the use of a non-scalable mechanism to deliver the results.
Instead of fixing this architectural limitation in the project, you've got behavior in here that prevents people from being able to rely on the tool to do the only thing that it's supposed to do well which is deliver test results.
Not only that, but it's an issue that doesn't show up so that when we test the system initially for proof of concept, it looks fine. The dealbreaking issue only crops up later.
This is how you tank a software project. By shoving a critical bug under the rug. What a shame.
I was starting to look into a way to make ANSI escape codes work in these results. However it's pretty clear now that if we can't even get rid of this truncation in the first place, it wouldn't be worth trying.
from junit-plugin.
I was just trying to give some historical background for why the plugin imposes caps on text block length: because the simplistic built-in storage system scales poorly, and it seemed preferable to lose some potentially helpful diagnostic information than to risk having the whole server crash because one user peeked at an unstable build.
Also understood. But the fact still remains that there is no middle ground where one can choose to have the full output for (only) failed tests but allow the truncation of passing results. Currently if one wants the full output on failed results one is required to take full output on passing results also.
So can we direct this ticket about adding that middle ground?
from junit-plugin.
Really, 105 characters?!
Also, how did it know that omitting those 105 characters would %$@# me the hardest? .. It really seems to be able to locate exactly the things I care about in order to surgically omit.
from junit-plugin.
I should add, and maybe this is a separate ticket, but it's super annoying that the tail
after the ...[truncated 135494 chars]...
is displayed literally instead of interpreting the \n
s:
21:40:01 DEBUG| [stdout] [1,0]<stdout>:Instance 1: starting format of SCM (dcpm:/mnt/daos1)
21:40:01 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:01.556730 instance_storage.go:107: /mnt/daos0 (dcpm) needs format: true
21:40:01 DEBUG| [stdout] [1,0]<stdout>:Instance 0: starting format of SCM (dcpm:/mnt/daos0)
21:40:12 DEBUG| [stdout] [1,3]<stdout>:Instance 0: finished format of SCM (dcpm:/mnt/daos0)
21:40:12 DEBUG| [stdout] [1,2]<stdout>:Instance 0: finished format of
...[truncated 135494 chars]...
g superblock\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DAOS I/O Server instance 1 storage ready\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:13.976631 instance_superblock.go:107: /mnt/daos1: checking superblock\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:13.977071 instance_superblock.go:111: /mnt/daos0: needs superblock (doesn\'t exist)\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:13.977124 instance_superblock.go:136: idx 0 createSuperblock()\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:13.977129 instance_superblock.go:111: /mnt/daos1: needs superblock (doesn\'t exist)\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:13.977195 instance_superblock.go:136: idx 1 createSuperblock()\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:13.977621 instance_superblock.go:166: creating /mnt/daos0/superblock: (rank: NilRank, uuid: a2d362ee-3d70-48b9-8240-95e881a84267)\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:13.977683 instance_superblock.go:166: creating /mnt/daos1/superblock: (rank: NilRank, uuid: 8c3bda34-7bc0-4b27-8f5e-f15d8f6ba682)\n21:40:13 DEBUG| [stdout] [1,0]<stdout>:DEBUG 21:40:13.978129 database.go:345: system db start: isReplica: true, isBootstrap: true
from junit-plugin.
Now that I look a bit more, this seems to be filed in Jenkins' Jira also.
Looking at the code, the trimming does not not appear to be conditional on the testing passing or failing:
junit-plugin/src/main/java/hudson/tasks/junit/CaseResult.java
Lines 150 to 152 in 5f7513b
There's no condition there for passing/failing.
from junit-plugin.
Ahhh. Looks like the code for handling the ...[truncated 135494 chars]...
is actually in
junit-plugin/src/main/java/hudson/tasks/junit/CaseResult.java
Lines 193 to 202 in 5f7513b
from junit-plugin.
The issue is that thereβs large performance impact of storing huge stack traces I believe, as they can be kept in memory and shown on pages
@jglick probably knows more
from junit-plugin.
Yes if you are using the built-in test storage,, giant test stdout/err is going to become giant CDATA
blocks in XML and giant String
s in Java heap and cause terrible performance problems loading the build record, as well as a lot of disk usage. Do not care so much about how long the HTML for that test result page is.
from junit-plugin.
Also: if you are in fact using another storage system, like an RDB π, it may be perfectly reasonable to just dump a few megs of text into a test result whether passing or failing: $JENKINS_HOME/builds/
would be no bigger, result summaries should be the same speed to load (I hope!), and the text should never be loaded into RAM unless and until you display that test case details page, and then promptly garbage-collected (again, needs to be verified).
from junit-plugin.
$JENKINS_HOME/builds/ would be no bigger
Extra disk use of (ideally only failing) builds to store the data needed to fix a bug is the least of my concern, and money well spent. Wasting disk space for the same on successful builds I am little less happy about.
But I am also concerned about RAM. If I am not using RDB (do you mean Ceph for this?) for storage and am just using the file system (i.e. the default storage mechanism) why is the impact on RAM different?
Is there anything specific about using RDB with Jenkins? Is there any documentation about it, why I would want to use it, etc.?
from junit-plugin.
money well spent. Wasting disk space for the same on successful builds I am little less happy about.
Understood. I was just trying to give some historical background for why the plugin imposes caps on text block length: because the simplistic built-in storage system scales poorly, and it seemed preferable to lose some potentially helpful diagnostic information than to risk having the whole server crash because one user peeked at an unstable build.
why is the impact on RAM different?
IIRC the whole junitResults.xml
file will be parsed into in-heap objects when using the default storage. There is no such thing as random access here (barring weird hacks like implementing a limited XML parser and saving byte offsets into the XML file). By contrast, when information about a particular test case can be obtained via a SQL query on demand, there is no need to load all of the data for all of the test results in the build into memory at once. (Again I am not sure offhand that the new storage actually does this properly, but it should if not.)
Is there anything specific about using RDB with Jenkins? Is there any documentation about it
Still experimental AFAIK; @timja can direct you to current docs.
from junit-plugin.
https://plugins.jenkins.io/junit-sql-storage/
IIRC the whole junitResults.xml file will be parsed into in-heap objects when using the default storage. There is no such thing as random access here (barring weird hacks like implementing a limited XML parser and saving byte offsets into the XML file). By contrast, when information about a particular test case can be obtained via a SQL query on demand, there is no need to load all of the data for all of the test results in the build into memory at once. (Again I am not sure offhand that the new storage actually does this properly, but it should if not.)
Certain parts are optimised, but if you ask for a suite result you get everything in the suite...
Pages like test results table will do that.
It should be better than before but may need more optimising.
The objects are very interlinked and context isn't passed around in the API as it's expected each child object has a link to it's parent that's already populated...
You can see the implemented APIs in this class:
https://github.com/jenkinsci/junit-plugin/blob/master/src/main/java/io/jenkins/plugins/junit/storage/TestResultImpl.java
from junit-plugin.
And update the current documentation to actually be what the plugin does, and that's to truncate all results, passing or failing, but just to different degrees.
from junit-plugin.
both is fine with me, contributions welcome. Especially documentation fixes
from junit-plugin.
This should be partially addressed by #601, which replaces the keepLongStdio
boolean property with stdioRetention
. This property can be set to one of three values:
none
- Long output from all tests is truncatedfailed
- Long output from failed tests is kept in its entirety, long output from passing tests is truncatedall
- Long output from all tests is kept in its entirety
Note that if your test output is stored exclusively in a separate file (e.g., if you're using Maven and you've configured it with redirectTestOutputToFile set to true
), then truncation may still take place. Specifically, if stdioRetention = 'all'
or stdioRetention = 'failed'
and the test has failed, then all files less than 1MB in size will be retained completely, but files 1MB and larger will be truncated and only the first and last 50KB will be retained. And, if stdioRetention = 'none'
or stdioRetention = 'failed'
and the test has passed, then files 1KB and larger will be truncated and only the first and last 500 bytes will be retained.
This matches the prior behavior for this scenario, and is done intentionally to avoid breaking environments where the plugin is already configured with keepLongStdio = true
(see CaseResult::possiblyTrimStdio from the commit directly before #601 was merged).
from junit-plugin.
Related Issues (20)
- check result is not reported if no tests are found HOT 1
- Dependency Dashboard
- Build failure when enforcing RequireUpperBoundDeps for kotlin
- Publish Junit Report throws java.io.FileNotFoundException: /opt/jdk/openJDK8/jre/lib/resources.jar HOT 1
- JavaScript browser console error: showFailuresLink is null HOT 7
- [failed-to-read] result.xml on Github
- make order of test execution obvious when viewing results HOT 1
- Allow grouping results by the "testsuite" tag
- Stacktrace after moving Junit results to external storage and cannot load graph of previous results
- In Permalinks, "Latest Test Result" field is not even shown for builds not completed in "lastCompletedBuild/testReport/" HOT 5
- Illegal XML characters in test output not handled correctly
- AggregatedTestResultPublisherTest fails with Jenkins 2.434 HOT 2
- Support multi-line failure/error messages
- Allow old test reports via configuration HOT 4
- Support multiple system-out tags per testcase HOT 2
- Pass FlowNode to TestDataPublisher HOT 1
- Conditionally disable "History" and "Test Result Trend"
- Doesn't handle ANSI colors HOT 2
- Environmental variable as junitxml path
- Failed to download 1265.v65b_14fa_f12f0 version HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from junit-plugin.