When I last tried Benchmark (which was a while ago;) I was able to run createScorecards.sh successfully on the Benchmark docker image, which is headless.
I've just tried this again but it failed - output given below.
I'd love to be able to run ZAP against Benchmark as part of our scheduled tests - this is a blocker :(
Analyzing results from Benchmark_1.2beta_ZAPweekly-18000.xml
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
at java.awt.Window.(Window.java:535)
at java.awt.Frame.(Frame.java:420)
at javax.swing.JFrame.(JFrame.java:218)
at org.owasp.benchmark.score.report.ScatterTools.display(ScatterTools.java:54)
at org.owasp.benchmark.score.report.ScatterTools.(ScatterTools.java:49)
at org.owasp.benchmark.score.report.Report.(Report.java:76)
at org.owasp.benchmark.score.BenchmarkScore.process(BenchmarkScore.java:406)
at org.owasp.benchmark.score.BenchmarkScore.main(BenchmarkScore.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Actual results file generated: /benchmark/benchmark/scorecard/Benchmark_v1.2beta_Scorecard_for_OWASP_ZAP_vD-2016-02-01.csv
Error processing results/Benchmark_1.2beta_ZAPweekly-18000.xml. Continuing.
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
at java.awt.Window.(Window.java:535)
at java.awt.Frame.(Frame.java:420)
at javax.swing.JFrame.(JFrame.java:218)
at org.owasp.benchmark.score.report.ScatterTools.display(ScatterTools.java:54)
at org.owasp.benchmark.score.report.ScatterTools.(ScatterTools.java:49)
at org.owasp.benchmark.score.report.Report.(Report.java:76)
at org.owasp.benchmark.score.BenchmarkScore.process(BenchmarkScore.java:406)
at org.owasp.benchmark.score.BenchmarkScore.main(BenchmarkScore.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Analyzing results from Benchmark_1.2beta-findbugs-v3.0.1-315.xml
Actual results file generated: /benchmark/benchmark/scorecard/Benchmark_v1.2beta_Scorecard_for_FindBugs_v3.0.1.csv
Error processing results/Benchmark_1.2beta-findbugs-v3.0.1-315.xml. Continuing.
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
at java.awt.Window.(Window.java:535)
at java.awt.Frame.(Frame.java:420)
at javax.swing.JFrame.(JFrame.java:218)
at org.owasp.benchmark.score.report.ScatterTools.display(ScatterTools.java:54)
at org.owasp.benchmark.score.report.ScatterTools.(ScatterTools.java:49)
at org.owasp.benchmark.score.report.Report.(Report.java:76)
at org.owasp.benchmark.score.BenchmarkScore.process(BenchmarkScore.java:406)
at org.owasp.benchmark.score.BenchmarkScore.main(BenchmarkScore.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Analyzing results from Benchmark_1.2beta-findsecbugs-v1.4.3-196.xml
Actual results file generated: /benchmark/benchmark/scorecard/Benchmark_v1.2beta_Scorecard_for_FBwFindSecBugs_v1.4.3.csv
Error processing results/Benchmark_1.2beta-findsecbugs-v1.4.3-196.xml. Continuing.
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
at java.awt.Window.(Window.java:535)
at java.awt.Frame.(Frame.java:420)
at javax.swing.JFrame.(JFrame.java:218)
at org.owasp.benchmark.score.report.ScatterTools.display(ScatterTools.java:54)
at org.owasp.benchmark.score.report.ScatterTools.(ScatterTools.java:49)
at org.owasp.benchmark.score.report.Report.(Report.java:76)
at org.owasp.benchmark.score.BenchmarkScore.process(BenchmarkScore.java:406)
at org.owasp.benchmark.score.BenchmarkScore.main(BenchmarkScore.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Analyzing results from Benchmark_1.2beta_ZAPweekly-18000.prev
Error!!: actual results were null for file: results/Benchmark_1.2beta_ZAPweekly-18000.prev
Tool scorecards computed.
Vulnerability scorecards computed.
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
at java.awt.Window.(Window.java:535)
at java.awt.Frame.(Frame.java:420)
at javax.swing.JFrame.(JFrame.java:218)
at org.owasp.benchmark.score.report.ScatterHome.display(ScatterHome.java:69)
at org.owasp.benchmark.score.report.ScatterHome.(ScatterHome.java:65)
at org.owasp.benchmark.score.report.ScatterHome.generateComparisonChart(ScatterHome.java:359)
at org.owasp.benchmark.score.BenchmarkScore.main(BenchmarkScore.java:353)
... 6 more
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:14 min
[INFO] Finished at: 2016-02-03T15:28:13+00:00
[INFO] Final Memory: 14M/265M
[INFO] ------------------------------------------------------------------------
[INFO] Build Time Summary:
[INFO]
[INFO] benchmark
[INFO] exec-maven-plugin:java (default) ......................... [6.397s]
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default) on project benchmark: An exception occured while executing the Java class. null: InvocationTargetException:
[ERROR] No X11 DISPLAY variable was set, but this program performed an operation which requires it.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException