Coder Social home page Coder Social logo

aqa-systemtest's Introduction

Eclipse Adoptium

This organization provides a home for Git repositories that contain the activities of the Adoptium Working Group, the Eclipse Adoptium Top Level Project and several Eclipse projects that fall under that top level project:

NOTE: The high-level project and issue tracking across all projects is kept in the Adoptium repo issue tracking system.


Please see the Eclipse Adoptium Project description for more information regarding the Adoptium top-level project or its sub-projects (visually depicted in the diagram below).

graph TD

subgraph Eclipse Adoptium
    classDef public fill:#CFE1F3,stroke:#333,stroke-width:4px,color:#000000
    classDef private fill:#FF0000,stroke:#333,stroke-width:4px,color:#000000
    style public fill:#CFE1F3,stroke:#333,stroke-width:4px,color:#000000
    style private fill:#FF0000,stroke:#333,stroke-width:4px,color:#000000
    subgraph Adoptium
        AdoptiumTrigger[adoptium]:::public --- website["adoptium.net"]:::public --- api["api.adoptium.net"]:::public --- blog["blog.adoptium.net"]:::public --- dash["dash.adoptium.net"]:::public
    end
    subgraph Temurin
        subgraph Build
            buildTrigger[temurin-build]:::public --- mirror["mirror-scripts"]:::public --- src["jdk, jdk8u, jdk8u-aarch32, jdk17u"]:::public --- release["github-release-scripts"]:::public --- binaries["temurin8-binaries,<br/>temurin11-binaries,<br/>temurin17-binaries,<br/>temurin19-binaries"]:::public --- installer["installer"]:::public --- build["build-jdk"]:::public
        end
        subgraph Infrastructure
            direction LR
            infraTrigger[infrastructure]:::public --- jenkins["ci-jenkins-pipelines"]:::public --- jenkinshelper["jenkins-helper"]:::public --- support["adoptium-support"]:::public
        end
    end
    subgraph Temurin Compliance
        TCKTrigger[temurin-compliance]:::private --- infra["infrastructure"]:::private --- jck8["JCK8-unzipped"]:::private --- jck11["JCK11-unzipped"]:::private --- jck17["JCK17-unzipped"]:::private --- jck19["JCK19-unzipped"]:::private
    end
    subgraph AQAvit
        AQAvitTrigger[aqa-tests]:::public --- tkg["TKG"]:::public --- test-tools["aqa-test-tools"]:::public --- stf["STF"]:::public --- systemtest["aqa-systemtest"]:::public --- bumblebench["bumblebench"]:::public --- run-aqa["run-aqa"]:::public
    end
    subgraph Incubator
        IncubatorTrigger[jdk11u-fast-startup-incubator]:::public
    end
end

Eclipse Adoptium Working Group

The Adoptium Working Group promotes and supports high-quality runtimes and associated technology for use across the Java ecosystem. Our vision is to meet the needs of Eclipse and the broader Java community by providing a marketplace for high-quality Java runtimes for Java-based applications. We embrace existing standards and a wide variety of hardware and cloud platforms.

Eclipse Adoptium Top Level Project

The mission of the Eclipse Adoptium Top-Level Project is to distribute high-quality runtimes and associated technology for use within the Java ecosystem. We achieve this through a set of Projects under the Adoptium Project Management Committee (PMC) and a close working partnership with external projects, most notably OpenJDK for providing the Java SE runtime implementation. Our goal is to meet the needs of both the Eclipse community and broader runtime users by providing a comprehensive set of technologies around runtimes for Java applications that operate alongside existing standards, infrastructures, and cloud platforms.

Eclipse AQAvit project

AQAvit is the quality and runtime branding evaluation project for Java SE runtimes and associated technology. During a release it takes a functionally complete Java runtime and ensures that all the additional qualities are present that make it suitable for production use. These quality criteria include good performance, exceptional security, resilience and endurance, and the ability to pass a wide variety of application test suites. In addition to verifying that functionally complete runtimes are release ready, the AQA tests may also serve to verify new functionality during runtime development.

Eclipse Temurin project

The Eclipse Temurin project provides code and processes that support the building of runtime binaries and associated technologies that are high performance, enterprise-caliber, cross-platform, open-source licensed, and Java SE TCK-tested for general use across the Java ecosystem.

Eclipse Temurin Compliance project

The Eclipse Temurin Compliance project is responsible for obtaining, managing, and executing the Oracle Java SE Compatibility Kit (JCK) on Eclipse Temurin binaries. The work is done on private infrastructure and using code managed in closed repositories only available to committers of Temurin Compliance. The public artefacts produced by this project are limited to an indication of whether a particular Eclipse Temurin binary is Java SE compliant or not.

Eclipse Mission Control project

Eclipse Mission Control enables you to monitor and manage Java applications without introducing the performance overhead normally associated with these types of tools. It uses data collected for normal adaptive dynamic optimization of the Java Virtual Machine (JVM). Besides minimizing the performance overhead, this approach eliminates the problem of the observer effect, which occurs when monitoring tools alter the execution characteristics of the system.

aqa-systemtest's People

Contributors

adamfarley avatar andrew-m-leonard avatar battleblow avatar divydeva avatar gdams avatar helenmasters avatar ibmjimmyk avatar jasonfengj9 avatar jiekang avatar joeyleeeeeee97 avatar karianna avatar keithc-ca avatar lasombra avatar llxia avatar longyuzhang avatar lumpfish avatar lutkerd avatar mamatha-jv avatar mesbah-alam avatar mikezhang1234567890 avatar navyxliu avatar pshipton avatar psoujany avatar say-droid427 avatar smlambert avatar sophia-guo avatar stevewallin avatar sxa avatar sxa555 avatar theresa-m avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aqa-systemtest's Issues

Remove build.xml from openjdk-systemtest root folder

We are no longer using the build.xml in openjdk-systemtest root folder that was invoking openjdk-test/build.xml. It was added for incorporating systemtest with testkitGen. However, since we are using the new folder structure for systemtest in openjdk-test repo, we no longer need this build file.

OpenJDK issue: ConcurrentLoadTest fails with OpenJDK 10

Overview:
ConcurrentLoadTest fails consistently, with OpenJDK 10, across all platforms.

Error:

  LT  testStarted : testAPI(net.adoptopenjdk.test.concurrent.ConcurrentSkipListMapTest)
        LT  testFinished: testAPI(net.adoptopenjdk.test.concurrent.ConcurrentSkipListMapTest)
        LT  testStarted : testStress(net.adoptopenjdk.test.concurrent.ConcurrentSkipListMapTest)
        LT  testFailure: testStress(net.adoptopenjdk.test.concurrent.ConcurrentSkipListMapTest): null
        LT  java.util.NoSuchElementException
        LT  	at java.base/java.util.concurrent.ConcurrentSkipListMap$KeyIterator.next(ConcurrentSkipListMap.java:2135)
        LT  	at net.adoptopenjdk.test.concurrent.ConcurrentSkipListMapTest.testStress(ConcurrentSkipListMapTest.java:343)
        LT  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        LT  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        LT  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        LT  	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        LT  	at junit.framework.TestCase.runTest(TestCase.java:176)
        LT  	at junit.framework.TestCase.runBare(TestCase.java:141)
        LT  	at junit.framework.TestResult$1.protect(TestResult.java:122)
        LT  	at junit.framework.TestResult.runProtected(TestResult.java:142)
        LT  	at junit.framework.TestResult.run(TestResult.java:125)
        LT  	at junit.framework.TestCase.run(TestCase.java:129)
        LT  	at junit.framework.TestSuite.runTest(TestSuite.java:252)
        LT  	at junit.framework.TestSuite.run(TestSuite.java:247)
        LT  	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
        LT  	at org.junit.runners.Suite.runChild(Suite.java:128)
        LT  	at org.junit.runners.Suite.runChild(Suite.java:27)
        LT  	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        LT  	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        LT  	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        LT  	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        LT  	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        LT  	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        LT  	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
        LT  	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
        LT  	at net.adoptopenjdk.loadTest.adaptors.JUnitAdaptor.executeTest(JUnitAdaptor.java:129)
        LT  	at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:177)
        LT  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        LT  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        LT  	at java.base/java.lang.Thread.run(Thread.java:844)
        LT  testFinished: testStress(net.adoptopenjdk.test.concurrent.ConcurrentSkipListMapTest)

Example Job link:
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk10_hs_systemtest_aarch64_linux/10/artifact/openjdk-tests/TestConfig/test_output_15211258652983/TestTargetResult.tap/*view*/

Note
This is not a test issue. The actual defects will be opened in OpenJDK community. This openjdk-systemtest issue is to be used only to track the test exclusion.

LockingLoadTest fails on hotspot VMs

LockingLoadTest fails on openjdk8_hs_systemtest_x86-64_linux with error:

 LLT testFailure: testPow(net.adoptopenjdk.test.math.MathAPITest): pow(double)[5] :: expected:<8.382703296667981E-131> but was:<8.382703296667982E-131>
        LLT junit.framework.AssertionFailedError: pow(double)[5] :: expected:<8.382703296667981E-131> but was:<8.382703296667982E-131>
        LL

Build link
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_hs_systemtest_x86-64_linux/lastSuccessfulBuild/artifact/openjdk-tests/TestConfig/test_output_15187805588474/TestTargetResult.tap/*view*/

JCK read me update

Under the section ""Executing the JCK tests via 'stf.pl'" we have listed "executiontype" and "withagent". We suggest removing them and just mention below

executiontype: The stf automation runs tests in multijvm way.

Enable users to run system tests via Jenkins

This issue follows on from #11

In order to investigate a system test failure a user may want to rerun the test with additional debug options. The user basically needs to execute a command on the test machine of the form
make test.Custom TEST=xxxx TEST_ARGS="xxxx" JAVA_ARGS="xxxx"
A Jenkins job with these Parameters:

  1. The repository containing the test to be run (openjdk-systemtest / openj9-systemtest)
  2. The value for TEST
  3. The value for TEST_ARGS
  4. The value for JAVA_ARGS
    The job would then need to:
  5. clone the system test repositories
  6. Build the repositories
  7. Run the
    make test.Custom TEST=xxxx TEST_ARGS="xxxx" JAVA_ARGS="xxxx"
    command line
  8. Retrieve the test output for examination by the user

Jdi tests failing with "Address already in use" failure on x86-64

Intermittent failure showing up in (at least) x64 jobs

BLIC Listen address[1] = localhost:30050 listening on port 30050:...... at Sep 15, 2017 4:39:50 PM
BLIC Listen address[0] = localhost:30060 listening on port 30060:...... at Sep 15, 2017 4:39:50 PM
BLIC IO Exception from Listening VM (Connectors.java) on port 30060 @Sep 15, 2017 4:39:50 PM
BLIC Errors in accepting connection on port 30060
BLIC Errors vm object is null using these connection settings port 30060
BLIC stderr java.net.BindException: Address already in use (Bind failed)
BLIC stderr 	at java.net.PlainSocketImpl.socketBind(Native Method)
BLIC stderr 	at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
BLIC stderr 	at java.net.ServerSocket.bind(ServerSocket.java:375)
BLIC stderr 	at java.net.ServerSocket.bind(ServerSocket.java:329)
BLIC stderr 	at com.sun.tools.jdi.SocketTransportService.startListening(SocketTransportService.java:255)
BLIC stderr 	at com.sun.tools.jdi.SocketTransportService.startListening(SocketTransportService.java:283)
BLIC stderr 	at com.sun.tools.jdi.GenericListeningConnector.startListening(GenericListeningConnector.java:113)
BLIC stderr 	at com.sun.tools.jdi.SocketListeningConnector.startListening(SocketListeningConnector.java:85)
BLIC stderr 	at com.sun.tools.jdi.GenericListeningConnector.accept(GenericListeningConnector.java:158)
BLIC stderr 	at net.adoptopenjdk.test.jdi.debugger.Connectors.createListeningVM(Connectors.java:561)
BLIC stderr 	at net.adoptopenjdk.test.jdi.debugger.Connectors$CreateConnectionThread.run(Connectors.java:718)
BLIC ...... VirtualMachine obtained on port 30050 at Sep 15, 2017 4:39:59 PM
BLIC No errors in accepting connection on port 30050
BLIC ***** Finished connecting ******
BLIC 
BLIC ***** ERROR: vm0 was null ******
BLI1 stderr ERROR: transport error 202: recv failed during handshake: Connection reset by peer
BLI1 stderr ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
BLI1 stderr JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:750]

Enable users to override the test results directory from make

It would be useful whilst running a specific test multiple times (for the purposes of defect recreation) to be able to override the test results directory at the level where the test is being executed,

e.g. something like
make test.Custom TEST=xxxx TEST_ARGS="xxxx" JAVA_ARGS="xxxx" RESULTS_ROOT="/tmp/stf-recreate-issue8/$datetime"

as otherwise both failing and passing test results get deleted indiscriminately when we have a predetermined number of results (10?).

I don't know if RESULTS_ROOT is the appropriate name for this macro.

Standardize the inconsistent logging, asserting, and println approach in JLM system tests

The reporting and error logging approach in JLM tests is inconsistent. Some of the tests generate "CSV" data files where some info is logged. The rest go into standard err and out. There are also some tests (e.g. VMLogger, where a log file is used to log detailed results of what the invoked Bean API's return. The output handling is not standardized in general.

This issue will track the work of standardizing the inconsistent logging, asserting, and println approach in JLM system tests

Create suitable .jti files for running the JCK10

As a tactical fudge I copied the JCK9 JTI files held in this repository into the place where they should be for JCK10. This is almost certainly not the right solution as it hasn't been modified for JCK10 but at least allows STF to start running against Java10. This issues should cover modifying the files in openjdk.test.jck/config's jck10 directory so they are suitably set up for JCK10.

(I'm running JCK10 from my branch which has the jck9 directory duplicated into jck10

JDITest failing on openjdk8_openj9 x86-64 Linux with JDWP option sysntax error

JDITest is failing on openjdk8_openj9 x86-64 Linux with JDWP option sysntax error as follows:

ASI1 stderr ERROR: JDWP option syntax error: -agentlib:jdwp=transport=dt_socket,address=30000,timeout=300000,trace=PROG+CMD+PACK+DATA

Build URL: https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_j9_systemtest_x86-64_linux/26/

JDITest is also failing on openjdk9_openj9 Linux x86-x64 build, but with a different error:

ERROR: transport error 202: recv failed during handshake: Connection reset by peer
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:730]

Build URL: https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk9_j9_systemtest_x86-64_linux/

Fix top.xml and remove hard coded value for ${java8_compiler}

Copying Lan's comment in #89 (review) as description of this PR:

We should not hard code ${java8_compiler}, what about java9, 10, 11?
I think the code works because in top.xml, it happens to set java8_compiler and java9_compiler to java.home if it is not defined.

For example, the output for jdk10 testing is very confusing:

17:55:24      [echo] java8_home set to /home/jenkins/workspace/openjdk10_j9_systemtest_x86-64_linux/openjdkbinary/j2sdk-image/bin/..
17:55:24      [echo] java8_compiler set to /home/jenkins/workspace/openjdk10_j9_systemtest_x86-64_linux/openjdkbinary/j2sdk-image/bin/javac

https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk10_j9_systemtest_x86-64_linux/6/consoleFull

top.xml should be fixed. There is no need to have separate variables for java compiler.

MauveSingleInvocationLoadTest fails on various platforms

openjdk10_hs_systemtest_x86-64_linux
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk10_hs_systemtest_x86-64_linux/

        LT  FAIL: gnu.testlet.javax.swing.DefaultListSelectionModel.setSelectionInterval: SINGLE_SELECTION_X (number 7)
        LT  got 2 but expected 4

openjdk9_hs_systemtest_x86-64_macos: && openjdk8_hs_systemtest_x86-64_macos
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk9_hs_systemtest_x86-64_macos/lastSuccessfulBuild/artifact/openjdk-tests/TestConfig/test_output_15196529222063/TestTargetResult.tap/*view*/

&&

https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_hs_systemtest_x86-64_macos/

 LT  java.lang.ClassNotFoundException: gnu.testlet.java.awt.color.ColorSpace.getInstance
        LT  	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
        LT  	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)
        LT  	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
        LT  	at java.base/java.lang.Class.forName0(Native Method)
        LT  	at java.base/java.lang.Class.forName(Class.java:292)
        LT  	at net.adoptopenjdk.loadTest.adaptors.MauveAdaptor.executeTest(MauveAdaptor.java:50)
        LT  	at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:177)
        LT  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        LT  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        LT  	at java.base/java.lang.Thread.run(Thread.java:844)

openjdk9_j9_systemtest_x86-64_linux
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk9_j9_systemtest_x86-64_linux/21/console

openjdk-build/makefile error when using Windows subst drive letters

Some of the JCK path names are too long for certain Windows commands to process. A way around this is the use the Windows 'subst' command to allocate a drive letter to the first part of the path. However, doing this seems to affect the gnu make wildcard function which is used to check the existence of the make input parameters.
It is not clear why wildcard does not work properly, but trial and error has shown that if backslashes in any paths are replaced with a single forward slash, the function works as is does for a unsubstituted path.
This issue proposes to fix the errors by replacing backslashes with forward slashes prior to running the wildcard function.

MauveSingleThreadLoadTest fails on various platforms

MauveSingleThreadLoadTest fails on openjdk8_j9_systemtest_x86-64_linux:
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_j9_systemtest_x86-64_linux/18/artifact/openjdk-tests/TestConfig/test_output_15187920382753/TestTargetResult.tap/*view*/

LT  >>> Captured test output >>>
        LT  PASS: gnu.testlet.java.math.BigInteger.multiply (number 0)
        LT  PASS: gnu.testlet.java.math.BigInteger.multiply (number 1)
        LT  PASS: gnu.testlet.java.math.BigInteger.multiply (number 2)
        LT  PASS: gnu.testlet.java.math.BigInteger.multiply (number 3)
        LT  PASS: gnu.testlet.java.math.BigInteger.multiply (number 4)
        LT  PASS: gnu.testlet.java.math.BigInteger.multiply (number 5)
        LT  PASS: gnu.testlet.java.math.BigInteger.multiply (number 6)
        LT  PASS: gnu.testlet.java.math.BigInteger.multiply (number 7)
        LT  FAIL: gnu.testlet.java.math.BigInteger.multiply (number 8)
        LT  <<<

Failure on openjdk9 hotspot Macos vm:
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk9_hs_systemtest_x86-64_macos/lastSuccessfulBuild/artifact/openjdk-tests/TestConfig/test_output_15196529222063/TestTargetResult.tap/*view*/

 LT  06:08:27.780 - First failure detected by thread: load-0. Not creating dumps as not running on an IBM JVM
        LT  06:08:27.809 - Test failed
        LT    Failure num.  = 1
        LT    Test number   = 43
        LT    Test details  = 'Mauve[gnu.testlet.java.awt.color.ColorSpace.getInstance]'
        LT    Suite number  = 0
        LT    Thread number = 0
        LT  >>> Captured test output >>>
        LT  Test failed:
        LT  java.lang.ClassNotFoundException: gnu.testlet.java.awt.color.ColorSpace.getInstance
        LT  	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
        LT  	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)
        LT  	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
        LT  	at java.base/java.lang.Class.forName0(Native Method)
        LT  	at java.base/java.lang.Class.forName(Class.java:292)
        LT  	at net.adoptopenjdk.loadTest.adaptors.MauveAdaptor.executeTest(MauveAdaptor.java:50)
        LT  	at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:177)
        LT  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        LT  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        LT  	at java.base/java.lang.Thread.run(Thread.java:844)
        LT  <<<
        LT  

Fails with Openjdk8 hotspot macos vm:
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_hs_systemtest_x86-64_macos/lastSuccessfulBuild/artifact/openjdk-tests/TestConfig/test_output_15194062996965/TestTargetResult.tap/*view*/

Fails on openjdk9 hotspot aarch64 linux: https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk9_hs_systemtest_aarch64_linux/24/console

Adding ability to run multiple subsets of JCK tests at once

For the convenience of executing JCK tests, there is a need to run arbitrary subsets of JCK tests (e.g. api/java_lang and api/java_math/BigDecimal).
Based on the description in openjdk.test.jck/include/test_targets.mk, there is a test.jck.custom target which can help developer to execute arbitrary subset of JCK tests. But it seems this target can only accept one test subset at once, if multiple subsets were provided, it will throw exception.

For example:

  1. cmd like this make -k test.jck.custom JCKVERSION=jck9 JCKTESTSUITE=RUNTIME JCKTEST="api/java_lang" will successfully generate the STF JCK test script and start running the api/java_lang JCK test.

  2. cmd like this make -k test.jck.custom JCKVERSION=jck9 JCKTESTSUITE=RUNTIME JCKTEST="api/java_lang api/java_math/BigDecimal" will failed the STF JCK test execution with error message

STF 18:06:36.920 - Redirecting stderr to /tmp/stf/20171109-180633-Jck/results/3.JCK.stderr
STF 18:06:36.920 - Redirecting stdout to /tmp/stf/20171109-180633-Jck/results/3.JCK.stdout
STF 18:06:36.933 - Monitoring processes: JCK
JCK stderr + set jck.env.runtime.testExecute.otherOpts "   --add-modules java.xml.ws.annotation,java.xml.bind,java.xml.ws,java.activation,java.corba -Xdump:system:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -Xdump:snap:none -Xdump:snap:events=gpf+abort+traceassert+corruptcache -Xdump:java:none -Xdump:java:events=gpf+abort+traceassert+corruptcache -Xdump:heap:none -Xdump:heap:events=gpf+abort+traceassert+corruptcache "
JCK stderr Configuration file can not be imported due to question with specified tag "jck.env.runtime.testExecute.otherOpts" was not found on path. The path is:jck.intro
JCK stderr jck.env.simpleOrAdvanced (advanced)
JCK stderr jck.env.envName (jck_runtime)
JCK stderr jck.env.description (JCK9 runtime template jti for ST...)
JCK stderr jck.env.platform (jre)
JCK stderr jck.tests.chooseTests (Yes)
JCK stderr jck.tests.treeOrFile (tree)
JCK stderr jck.tests.tests (api/java_lang api/java_math/BigD...)
JCK stderr jck.env.testPlatform.intro
JCK stderr jck.env.testPlatform.nativeCode (Yes)
JCK stderr jck.env.testPlatform.multiJVM (Yes)
JCK stderr jck.env.testPlatform.useAgent (No)
JCK stderr jck.env.testPlatform.os (Current system)
JCK stderr jck.env.runtime.intro
JCK stderr jck.env.runtime.otherJVM
JCK stderr jck.env.runtime.testExecute.intro
JCK stderr jck.env.runtime.testExecute.cmdAsFile (/java9/bin/java)
JCK stderr jck.env.runtime.testExecute.classpath (command line option)
JCK stderr jck.env.runtime.testExecute.classpathOpt (-classpath #)
JCK stderr jck.env.runtime.testExecute.additionalClasspath ()
JCK stderr jck.env.moduleSystem.vmOptions (vmAddModsOptionTemplate=--add-mo...)
JCK stderr jck.env.runtime.testExecute.nativeLibsLinkage (dynamic)
JCK stderr jck.env.runtime.testExecute.libPath (environment variable)
JCK stderr jck.env.runtime.testExecute.libPathEnv (will_be_set_by_test_automation_a...)
JCK stderr jck.env.runtime.testExecute.nativeLibrariesLocation (Yes)
JCK stderr jck.env.runtime.testExecute.nativeLibPathFileValue (will_be_set_by_test_automation_a...)
JCK stderr jck.env.runtime.testExecute.cannot_read
JCK stderr 
STF 18:06:38.599 - **FAILED** Process JCK ended with exit code (3) and not the expected exit code/s (0)
STF 18:06:38.599 - Monitoring Report Summary:
STF 18:06:38.599 -   o Process JCK ended with exit code (3) and not the expected exit code/s (0)
STF 18:06:38.600 - Killing processes: JCK
STF 18:06:38.600 -   o Process JCK is not running
**FAILED** at step 3 (Running JCK in multijvm way with Agent off). Expected return value=0 Actual=1 at /tmp/stf/20171109-180633-Jck/execute.pl line 93.
STF 18:06:38.619 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 18:06:38.620 - 
STF 18:06:38.620 - ====================   T E A R D O W N   ====================

Although we can use a loop (in script) to run a list of subsets, the test summary outputs will be separated and hard to track which tests passed and which tests failed. I think we need the STF JCK test to be able to run multiple subsets of JCK test at once, as we want the final test summary to be combined together.

JCK automation does not seem to specify a path to an OpenJDK reference VM implementation

From the JCK User's Guide:
It is important to note how the JCK compiler tests work. RMI compilers and the
Java programming language compiler (Java compiler) compile test files and the
resulting class files are executed on a Oracle reference runtime implementation.

The "reference VM" is to verify the "byte codes" produced by the javac compiler are the ones expected by a reference implementation for obvious portability reasons.

I do not see where the "reference VM" is specified in the JCK automation?

Add support for arm platforms (32 and 64 bit)

Failing to compile due to unrecognized platform: aarch64

https://ci.adoptopenjdk.net/view/work%20in%20progress/job/openjdk9_test_arm64_linux/10/console

18:07:41 /home/jenkins/workspace/openjdk9_test_arm64_linux/openjdk-test/TestConfig/scripts/build_test.xml:75: The following error occurred while executing this line:
18:07:41 /home/jenkins/workspace/openjdk9_test_arm64_linux/openjdk-test/systemtest/build.xml:73: The following error occurred while executing this line:
18:07:41 /home/jenkins/workspace/openjdk9_test_arm64_linux/openjdk-test/systemtest/openjdk-systemtest/openjdk.build/include/top.xml:64: The following error occurred while executing this line:
18:07:41 /home/jenkins/workspace/openjdk9_test_arm64_linux/openjdk-test/systemtest/openjdk-systemtest/openjdk.build/include/top.xml:460: The following error occurred while executing this line:
18:07:41 /home/jenkins/workspace/openjdk9_test_arm64_linux/openjdk-test/systemtest/openjdk-systemtest/openjdk.build/include/top.xml:551: java os.arch property value aarch64 not recognised. Update the get-platform-arch macrodef in openjdk.build/include/top.xml
18:07:41

SharedClassesWorkloadTest_Softmx_Increase_JitAot test fails on ppc64le linux & s390x linux due to not enough AOT being generated

Overview
SharedClassesWorkloadTest_Softmx_Increase_JitAot test fails on openjdk8_j9_systemtest_ppc64le_linux due to not enough AOT being generated

The test fails in step 10 below:

1  setUp    mkdir             Create the cache directory
2  setUp    java              Destroy Persistent Shared Classes Caches
3  setUp    java              Destroy Non-Persistent Shared Classes Caches
4  execute  Run java          Run Jvm1 workload process
5  execute  Run java          Run Jvm2 workload process
7  execute  java              List all caches
8  execute  java              Print Shared Classes Cache Stats
9  execute  Run java          Combine stderr files of jvm1 and jvm2
10  execute  count            Make sure the space for AOT data in shared cache is full

Build Links:
https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_j9_systemtest_ppc64le_linux/34/

Note:
These tests are hard to write. They contain potential for inconsistent behavior like this. They may need to be updated occasionally based on changes made to the implementation of JIT.

For this issue, we need to update the test so that enough AOT gets generated on ppc64le Linux.

Modularity tests break JCK builds

The addition of the new system tests for modularity have broken compilation of the JCK test jobs.

Error
[exec] makefile:46: *** "The variable PLATFORM needs to be defined"

Note:
We know currently JCKs use STF, and currently you can't get/compile STF without getting and compiling systemtests - we should open and issue and eventually fix this circular dependency.

mauve tests setSelectionInterval and setSelectionMode fail on jdk10 + jdk11

Mauve tests gnu.testlet.javax.swing.DefaultListSelectionModel.setSelectionInterval and gnu.testlet.javax.swing.DefaultListSelectionMode fail on jdk10 but pass on jdk8 and jdk9.

This may be because of a tightening of the specification when switching from a less restrictive mode to a more restrictive mode - see https://bugs.openjdk.java.net/browse/JDK-8191436. This issue is to investigate whether that is indeed the cause, and if so, exclude the tests from the mauve workloads, otherwise raise a bug report against openjdk.

The test is run as part of the SingleInvocationMauveTest and SingleThreadMauveTest load test workloads.

To view, run or change the individual test cases:

  1. Build the openjdk-systemtest repository. The make configure step will download and build the mauve test cases under a systemtest_prereqs directory

  • unzip the systemtest_prereqs/mauve/mauve.jar file
  • edit the test case
  • javac individual test cases - e.g.
    cd /home/user/systemtest_prereqs/mauve//gnu/testlet/javax/swing/DefaultListSelectionModel
    javac -cp /home/user/systemtest_prereqs/mauve setSelectionInterval.java

Open-source JLM SVT test project into Adopt

openjdk.test.jlm - should go to AdoptOpenJDK/openjdk-systemtest repo.
openj9.test.jlm - should go to eclipse/openj9-systemtest repo (need to checkout openjdk.test.jlm project during build of this project, as it contains parent classes being extended in openj9.test.jlm).

Multiple duplicate test results are generated within one test run

When executing multiple jck subsets in one test run, for example:

make -k test.jck.custom JCKVERSION=jck9 JCKTESTSUITE=RUNTIME 'JCKTEST=<multiple_test_subsets>'

Multiple test results folder are generated, and some test results folder will include the same test result.

drwxr-xr-x@ 14  staff   448B 17 Jan 10:45 20180110-142044-Jck
drwxr-xr-x@ 14  staff   448B 17 Jan 10:44 20180110-152032-Jck
drwxr-xr-x@ 14  staff   448B 17 Jan 10:44 20180110-165438-Jck
drwxr-xr-x@ 13  staff   416B 17 Jan 09:21 20180110-233837-Jck
drwxr-xr-x@ 13  staff   416B 17 Jan 09:21 20180111-153415-Jck
drwxr-xr-x@ 13  staff   416B 17 Jan 09:21 20180111-233821-Jck
drwxr-xr-x@ 13  staff   416B 17 Jan 09:21 20180115-161311-Jck
drwxr-xr-x@ 14  staff   448B 17 Jan 10:01 20180116-110859-Jck
drwxr-xr-x@ 14  staff   448B 17 Jan 09:24 20180116-221554-Jck

The reason why we run multiple test subset together is that we want to have a final result in the end of the test execution showing which tests were failed and which tests were passed. If those test subsets were executed separately, their test results will be separated and hard to trace.

So we run multiple test subsets in one run to get the results merged togertger. But the behaviour of generating test results folders changed, is this an expected behaviour? and do we have options to ask stf to generate one test result folder instead of multiples?

HCRLateAttach throwing AgentInitializationException on Java9

Issue shows on both the HotSpot builds and OpenJ9 across all architectures. Issue is in the asm dependency which currently has a beta version 6 that still shows an issue. Further investigation required and to monitor the asm project for a full v6 release

LT  stderr Caused by: java.lang.IllegalArgumentException
LT  stderr      at org.objectweb.asm.ClassReader.<init>(Unknown Source)
LT  stderr      at org.objectweb.asm.ClassReader.<init>(Unknown Source)
LT  stderr      at net.adoptopenjdk.test.hcrAgent.agent.StringTransformer.transform(StringTransformer.java:70)
LT  stderr      ... 6 more
LT  stderr Agent failed to start!
LT  stderr JVMJ9TI064E Agent initialization function Agent_OnAttach failed for library instrument, return code 102
LT  19:28:18.488 - Parsing inventory file. Root=/tmp/stf/20170916-192804-HCRLateAttachWorkload/results/1.LT.inventory File=openjdk.test.load/config/inventories/util/util.xml
AG  stderr Exception in thread "main" com.sun.tools.attach.AgentInitializationException: ATTACH_ERR AgentInitializationException102
AG  stderr      at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.parseResponse(jdk.attach@9-internal/OpenJ9VirtualMachine.java:324)
AG  stderr      at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.loadAgent(jdk.attach@9-internal/OpenJ9VirtualMachine.java:229)
AG  stderr      at net.adoptopenjdk.test.hcrAgent.agent.Attacher.main(Attacher.java:75)
STF 19:28:18.500 - **FAILED** Process AG  ended with exit code (1) and not the expected exit code/s (0)

Enable ability to run all tests in a JCK test suite

Example: I want to run all of the DEVTOOLS suite. There doesn't appear to be a suitable target for this, or if there is it's not documented in the make help.jck output.

Suggest make test.jck8b.runtime or make test.jck8b.runtime.all.

Migrate the project from Ant to Multimodule project with Maven

I am trying to use the project in an embedded environment and I would like to use this project to test all features from a Java Runtime Image but the project requires compilation in the destination.

I would like to propose to migrate Ant to Maven to offer more global consistency.

What is your opinion?

Juan Antonio

DirectByteBufferTest fails with NullPointerException on openjdk8_openj9 SDK

DirectByteBufferTest fails with NullPointerException on openjdk8_openj9 SDKs on, at least, x64 and ppc64le linux platforms. It works fine on hotspot, as well as openjdk9_openj9 SDKs.

Testcase

The test creates a array of ByteBuffer of length = 90, and allocates a new bytebuffer of ALLOC_SIZE=10 MBytes in each index of that array (by calling ByteBuffer.allocateDirect(..)) and de-refences immediately to allow GC to happen.
`public static void testDirectMemory() {
try {
int ITERATIONS = 90;
int ALLOC_SIZE = 10000000; // 10Mbytes

		ByteBuffer[] bbStore = new ByteBuffer[ITERATIONS];
		Runtime r = Runtime.getRuntime();

		System.out.println("Total memory is: " + r.totalMemory());
		System.out.println("Initial free memory: " + r.freeMemory());
		int i;
		for (i = 0; i < ITERATIONS; i++) {

			System.out.println("Iteration: " + i + " Total memory is: "
					+ r.totalMemory() + "Initial free memory: "
					+ r.freeMemory());
			bbStore[i] = ByteBuffer.allocateDirect(ALLOC_SIZE);
			// *** dereference to allow GC
			bbStore[i] = null;
		}
		
		assertTrue(i==ITERATIONS);
		
	} catch (Exception e) {
		System.out.println("Caught Exception: \n" + e);
		fail("Exception::"+e);
	}
}`

Error stacktrace

14:32:27 DBLT Caught Exception:
14:32:27 DBLT java.lang.NullPointerException
14:32:27 DBLT testFailure: testDirectMemory(net.adoptopenjdk.test.gc.directbytebuffer.DirectMemoryTest): Exception::java.lang.NullPointerException
14:32:27 DBLT junit.framework.AssertionFailedError: Exception::java.lang.NullPointerException
14:32:27 DBLT at junit.framework.Assert.fail(Assert.java:57)
14:32:27 DBLT at junit.framework.TestCase.fail(TestCase.java:227)
14:32:27 DBLT at net.adoptopenjdk.test.gc.directbytebuffer.DirectMemoryTest.testDirectMemory(DirectMemoryTest.java:49)
14:32:27 DBLT at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
14:32:27 DBLT at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
14:32:27 DBLT at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
14:32:27 DBLT at java.lang.reflect.Method.invoke(Method.java:498)
14:32:27 DBLT at junit.framework.TestCase.runTest(TestCase.java:176)
14:32:27 DBLT at junit.framework.TestCase.runBare(TestCase.java:141)
14:32:27 DBLT at junit.framework.TestResult$1.protect(TestResult.java:122)
14:32:27 DBLT at junit.framework.TestResult.runProtected(TestResult.java:142)
14:32:27 DBLT at junit.framework.TestResult.run(TestResult.java:125)
14:32:27 DBLT at junit.framework.TestCase.run(TestCase.java:129)
14:32:27 DBLT at junit.framework.TestSuite.runTest(TestSuite.java:252)
14:32:27 DBLT at junit.framework.TestSuite.run(TestSuite.java:247)
14:32:27 DBLT at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
14:32:27 DBLT at org.junit.runners.Suite.runChild(Suite.java:128)
14:32:27 DBLT at org.junit.runners.Suite.runChild(Suite.java:27)
14:32:27 DBLT at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
14:32:27 DBLT at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
14:32:27 DBLT at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
14:32:27 DBLT at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
14:32:27 DBLT at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
14:32:27 DBLT at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
14:32:27 DBLT at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
14:32:27 DBLT at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
14:32:27 DBLT at net.adoptopenjdk.loadTest.adaptors.JUnitAdaptor.executeTest(JUnitAdaptor.java:129)
14:32:27 DBLT at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:177)
14:32:27 DBLT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
14:32:27 DBLT at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
14:32:27 DBLT at java.lang.Thread.run(Thread.java:811)
14:32:27 DBLT testFinished: testDirectMemory(net.adoptopenjdk.test.gc.directbytebuffer.DirectMemoryTest)

Build

  1. openjdk8_j9_systemtest_x86-64_linux : https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_j9_systemtest_x86-64_linux/5/
  2. openjdk8_j9_systemtest_ppc64le_linux: https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_j9_systemtest_ppc64le_linux/3/

openjdk-systemtest failing with stf.pl not found in jvmtest in ci.adopt

We have builds failing in systemtest in ci.adopt due to stf.pl not found under jvmtest folder in the build. e.g, https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk10_hs_systemtest_x86-64_linux/lastSuccessfulBuild/artifact/openjdk-tests/TestConfig/test_output_15186193597840/TestTargetResult.tap/*view*/

This is due to the fact that, with the new change in TestKitGen being used from openj9, there is no JAVA_VERSION in the path. We need to update the systemtest build script and remove JAVA_VERSION.

MathLoadTest+LockingLoadTest failing on HotSpot VMs (8 and 9) on x86

Issue as follows - need to determine correct behaviour and decide which VM is giving the correct result, and adjust test accordingly. Currently test is set up with the expected result from J9

MLT testStarted : testExp(net.adoptopenjdk.test.math.MathAPITest)
MLT testFailure: testExp(net.adoptopenjdk.test.math.MathAPITest): exp(double)[9] :: expected:<11.3842406513381> but was:<11.384240651338098>
MLT junit.framework.AssertionFailedError: exp(double)[9] :: expected:<11.3842406513381> but was:<11.384240651338098>
MLT     at junit.framework.Assert.fail(Assert.java:57)

Standalone test case that should show the problem:

import java.math.*;

public class Exp
{
 public static void main(String[] args)
 {
   String rc_String;
   rc_String = Double.toString(Math.exp(-1.2232D));
   if ( rc_String.equals("0.2942869403572507") ) {
     System.out.println ("Test 1 passed: Expected 0.2942869403572507, got " + rc_String);
   }
   else {
     System.out.println ("Test 1 failed: Expected 0.2942869403572507 (value returned by OpenJDK), got " + rc_String);
   }
 }
}

Mauve Load tests failing on ppc64le with UnknownHostException

MauveSingleInvocationLoadTest and MauveSingleThreadLoadTest are failing across all versions of Java on ppc64le platforms.

Error

java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
14:58:09 LT at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
14:58:09 LT at java.base/java.lang.Thread.run(Thread.java:844)
14:58:09 LT Caused by: java.net.UnknownHostException: test-ubuntu-16-04-1: Name or service not known
14:58:09 LT at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
14:58:09 LT at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:924)
14:58:09 LT at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1504)
14:58:09 LT at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:843)
14:58:09 LT at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1494)
14:58:09 LT at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1626)
14:58:09 LT ... 11 more

Build links:

  1. openjdk9_hs_systemtest_ppc64le_linux: https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk9_hs_systemtest_ppc64le_linux/1/
  2. openjdk8_hs_systemtest_ppc64le_linux: https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_hs_systemtest_ppc64le_linux/1/
  3. openjdk9_j9_systemtest_ppc64le_linux: https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk9_j9_systemtest_ppc64le_linux/1/
    4.openjdk8_j9_systemtest_ppc64le_linux: https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk8_j9_systemtest_ppc64le_linux/1/

Add TEST_ARGS to make test command

Currently when running e.g. make test.JdiLoadTest I can't specify an individual test to run e.g. simple_launch. To do that I have to go right down to the perl layer underneath.

As I'm trying to investigate a failure in one test within the test suite adding this feature would make it easier to reproduce failures, as we could then allow the launching of single tests through jenkins directly. In effect I'm requesting the addition of TEST_ARGS="test=basic_launch" or something similar to the makefile.

Add initial set of mode specific jvm options to systemtest

The initial set of modes preferred by the jit team is : 100, 101, 103, 107, 107-OSRG, 110, 110-CS, 112, 121, 150, 150-CS, 188, 304, 557, 607, 607-OSRG, 615, 687, 688, 707.

We'd want to add the jvm options corresponding to the above modes to the systemtest playlist.

Add support for ARM platform

Attempting to run the tests on arm fails with

GEN stderr Caused by: net.adoptopenjdk.stf.StfException: Unknown architecture value: 'arm'. Expected one of '390|x86|ppc|x86'

java.lang.IllegalArgumentException: Constant pool index out of bounds

When there are a lot of custom annotations on a class, and later you want to read the annotations from the class file, the following happens (Open J9 JDK9, JDK10, linux x64):

java.lang.IllegalArgumentException: Constant pool index out of bounds
at sun.reflect.annotation.AnnotationParser.parseAnnotations(java.base@10-adoptopenjdk/AnnotationParser.java:77)
at java.lang.Class.getAnnotationCache(java.base@10-adoptopenjdk/Class.java:2646)
at java.lang.Class.getAnnotation(java.base@10-adoptopenjdk/Class.java:2294)

I'd provide an example if needed (this would be quite a bit of work though, since the annotations are compiler generated), however, I'm not even sure this is the right place to post this.

This used to work in Open j9 jdk8.

Network issues on Linux/s390x machine build-marist-s390x-sles-12

Various tests are failing on this machine. Haven't looked into it in detail yet but I suspect there's a defective /etc/hosts entry pointing the hostname to the wrong IP based on these exceptions

DBLT EchoServer Using channel group - sun.nio.ch.EPollPort@ac6dd7f
DBLT Completed Asynchronous Echo Server startup - available at /148.100.110.56:38637
DBLT testStarted : testMultipleSequentialConnect(net.adoptopenjdk.test.nio2.asyncio.client.MultipleConnectAsyncTest)
DBLT MultipleConnectAsyncTest.testMultipleSequentialConnect() creating 9 connections
DBLT testFailure: testMultipleSequentialConnect(net.adoptopenjdk.test.nio2.asyncio.client.MultipleConnectAsyncTest): [Failed Assertion Exceptions:
DBLT net.adoptopenjdk.test.nio2.util.AssertionFailedException: java.net.NoRouteToHostException: No route to host
DBLT    at net.adoptopenjdk.test.nio2.asyncio.client.MultipleConnectAsyncTest$1.failed(MultipleConnectAsyncTest.java:124)
DBLT    at net.adoptopenjdk.test.nio2.asyncio.client.MultipleConnectAsyncTest$1.failed(MultipleConnectAsyncTest.java:84)
DBLT    at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129)
DBLT    at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
DBLT    at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
DBLT    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
DBLT    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
DBLT    at java.base/java.lang.Thread.run(Thread.java:844)
DBLT    at java.base/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:122)
DBLT Caused by: java.net.NoRouteToHostException: No route to host
DBLT    at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method)
DBLT    at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:251)
DBLT    at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:197)
DBLT    at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:212)
DBLT    at java.base/sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293)
DBLT    ... 2 more
DBLT ---
LT  13:00:23.483 - Test failed
LT    Failure num.  = 1
LT    Test number   = 3562
LT    Test details  = 'Mauve[gnu.testlet.java.net.DatagramPacket.DatagramPacketTest2]'
LT    Suite number  = 0
LT    Thread number = 0
LT  >>> Captured test output >>>
LT  Test failed:
LT  java.net.UnknownHostException: openjdk-sles12: openjdk-sles12: Name or service not known
LT      at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1631)
LT      at gnu.testlet.java.net.DatagramPacket.DatagramPacketTest2.<init>(DatagramPacketTest2.java:70)
LT      at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
LT      at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
LT      at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
LT      at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
LT      at java.base/java.lang.Class.newInstance(Class.java:558)
LT      at net.adoptopenjdk.loadTest.adaptors.MauveAdaptor.executeTest(MauveAdaptor.java:51)
LT      at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:177)
LT      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
LT      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
LT      at java.base/java.lang.Thread.run(Thread.java:844)
LT  Caused by: java.net.UnknownHostException: openjdk-sles12: Name or service not known
LT      at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
LT      at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:924)
LT      at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1504)
LT      at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:843)  
LT      at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1494)
LT      at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1626)
LT      ... 11 more
LT  <<<

STF_COMMAND should be able to append extra STF perl options

the STF_COMMAND defined in the x.build/makefile should be able to append extra STF perl options.

For example, the -results-root can only be added when tests are executed from perl script. If a user want to define extra results-root dir, he have to use perl script to run the test. With the ability of appending extra perl script options, we can add extra options to test execution.

A proposal is changing
STF_COMMAND:=perl $(STF_ROOT)$(D)stf.core$(D)scripts$(D)stf.pl $(JAVA_ARGS_ARG) -test-root="$(STF_ROOT);$(OPENJDK_SYSTEMTEST_TARGET_ROOT)" -systemtest-prereqs="$(RESOLVED_PREREQS_ROOT)"

to

STF_COMMAND:=perl $(STF_ROOT)$(D)stf.core$(D)scripts$(D)stf.pl $(JAVA_ARGS_ARG) -test-root="$(STF_ROOT);$(OPENJDK_SYSTEMTEST_TARGET_ROOT)" -systemtest-prereqs="$(RESOLVED_PREREQS_ROOT)" $(EXTRA_STF_OPTION)

JCK test output is very verbose

JCK test output is very verbose. It is easily 7M. JCK tests prints out the progress of file downloading and cvs checkout. Could we use -q and only print error when the process fail, so that we can focus on the real test output?

Output:

23:14:21 configure-ant-1.10.1:
23:14:21      [echo] Executing macro download-file
23:14:21      [echo] srcurl is http://www-eu.apache.org/dist//ant/binaries/apache-ant-1.10.1-bin.zip
23:14:21      [echo] destdir is /tmp/apache-ant-1.10.1
23:14:21      [echo] destfile is apache-ant-1.10.1-bin.zip
23:14:21     [mkdir] Created dir: /tmp/apache-ant-1.10.1
23:14:21      [exec] --2017-11-12 23:14:20--  http://www-eu.apache.org/dist//ant/binaries/apache-ant-1.10.1-bin.zip
23:14:21      [exec] Resolving www-eu.apache.org (www-eu.apache.org)... 195.154.151.36, 2001:bc8:2142:300::
23:14:21      [exec] Connecting to www-eu.apache.org (www-eu.apache.org)|195.154.151.36|:80... connected.
23:14:21      [exec] HTTP request sent, awaiting response... 200 OK
23:14:21      [exec] Length: 8824128 (8.4M) [application/zip]
23:14:21      [exec] Saving to: '/tmp/apache-ant-1.10.1/apache-ant-1.10.1-bin.zip'
23:14:21      [exec] 
23:14:21      [exec]      0K .......... .......... .......... .......... ..........  0%  249K 34s
23:14:21      [exec]     50K .......... .......... .......... .......... ..........  1%  498K 26s
23:14:21      [exec]    100K .......... .......... .......... .......... ..........  1% 48.2M 17s
23:14:21      [exec]    150K .......... .......... .......... .......... ..........  2% 46.7M 13s
23:14:21      [exec]    200K .......... .......... .......... .......... ..........  2%  505K 13s
23:14:21      [exec]    250K .......... .......... .......... .......... ..........  3% 34.1M 11s
23:14:21      [exec]    300K .......... .......... .......... .......... ..........  4% 40.9M 10s
23:14:21      [exec]    350K .......... .......... .......... .......... ..........  4%  515K 10s
23:14:21      [exec]    400K .......... .......... .......... .......... ..........  5% 40.1M 9s
23:14:21      [exec]    450K .......... .......... .......... .......... ..........  5% 41.8M 8s
23:14:21      [exec]    500K .......... .......... .......... .......... ..........  6% 38.9M 7s
23:14:21      [exec]    550K .......... .......... .......... .......... ..........  6%  514K 8s
23:14:21      [exec]    600K .......... .......... .......... .......... ..........  7% 47.1M 7s
23:14:21      [exec]    650K .......... .......... .......... .......... ..........  8% 42.5M 7s
23:14:22      [exec]    700K .......... .......... .......... .......... ..........  8%  515K 7s
23:14:22      [exec]    750K .......... .......... .......... .......... ..........  9% 29.2M 7s
23:14:22      [exec]    800K .......... .......... .......... .......... ..........  9% 39.3M 6s
23:14:22      [exec]    850K .......... .......... .......... .......... .......... 10% 34.0M 6s
23:14:22      [exec]    900K .......... .......... .......... .......... .......... 11%  519K 6s
...
13:08:00 get-source:
13:08:00      [exec] Didn't find password for CVSROOT ':pserver:[email protected]:/cvs/mauve'.
13:08:01      [exec] U mauve/.classpath
13:08:01      [exec] cvs export: Updating mauve
13:08:01      [exec] U mauve/.cvsignore
13:08:01      [exec] U mauve/.project
13:08:01      [exec] U mauve/COPYING
13:08:01      [exec] U mauve/ChangeLog
13:08:02      [exec] U mauve/Harness.java
13:08:02      [exec] U mauve/Makefile.am
13:08:02      [exec] U mauve/Makefile.in
13:08:02      [exec] U mauve/README
13:08:02      [exec] U mauve/README.PKITS
13:08:02      [exec] U mauve/RunnerProcess.java
13:08:02      [exec] U mauve/THANKS
13:08:02      [exec] U mauve/acinclude.m4
13:08:02      [exec] U mauve/aclocal.m4
13:08:02      [exec] U mauve/config.guess
13:08:02      [exec] U mauve/config.sub
13:08:02      [exec] U mauve/configure
13:08:02      [exec] U mauve/configure.in
13:08:02      [exec] U mauve/install-sh
13:08:02      [exec] cvs export: Updating mauve/.externalToolBuilders
...

Malformed \uxxxx encoding error during make configure

On my laptop, running make configure resulted in the below error.

c:\git\openjdk-systemtest\openjdk.build\include\top.xml:64: The following error occurred while executing this line:
c:\git\openjdk-systemtest\openjdk.build\include\top.xml:447: java.lang.IllegalArgumentException: Malformed \uxxxx encoding.
at java.util.Properties.loadConvert(Properties.java:618)
at java.util.Properties.load0(Properties.java:435)
at java.util.Properties.load(Properties.java:364)
at org.apache.tools.ant.taskdefs.LoadProperties.execute(LoadProperties.java:199)
.
.

In top.xml, in the "set-platform-properties" macrodefinition, output of the command "java -XshowSettings:properties -version" is redirected into a property file. In the next step, using ant's loadproperties, the property file is read.

The issue is because of "\u" character appearing in the PATH. I had an entry in PATH env variable which is like "C:\stf\unzip". Since "\u" is coming up in the path "C:\stf**\u**nzip", ant loadproperties fails with the mentioned error.

Path entries in the generated property file appears with single "" slash. Solution to fix this problem is to change it to "" slash.

Enable JCK non-interactive test execution using STF

Most of the JCK tests can be executed from a command line.
Some of the tests require configuration parameters which correspond to the machine on which the tests are executing.
When run in this (command line) way, the tests write their status (Pass / Fail and other data) to text files, which the user then interrogates to determine whether the run was successful.

These steps can be automated using STF:

  1. Depending on the specific test executed, set configuration overrides for the machine on which the test is executing (overrides such as the location of the Java under test or the hostname of the test machine).
  2. Execute the required JCK command line.
  3. Filter the tests results files for errors. Pass the test run if no errors are found, otherwise fail.

The goal of the automation is to enable as much execution of as much of the JCK as possible on any test machine with access to the JCK with minimal additional configuration.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.