ga4gh / compliance Goto Github PK

We've found that the names in test report stack traces sometimes contain doubled characters.

The location of the doubled character seems random.

Here are a couple of examples from the same run.

In this example, see "ggetCallSetWithValidIDShouldSucceed", which should be "getCallSetWithValidIDShouldSucceed".

org.ga4gh.ctk.transport.GAWrapperException: Internal Server Error
at org.ga4gh.ctk.transport.avrojson.AvroJson.updateTheRespAndLogMessages(AvroJson.java::281)
at org.ga4gh.ctk.transport.avrojson.AvroJson.doGetResp(AvroJson.java:337)
at org.ga4gh.ctk.transport.avrojson.AvroJson.doGetResp(AvroJson.java:317)
at org.ga4gh.ctk.transport.protocols.Client$Variants.getCallSet(Client.java:211)
at org.ga4gh.cts.api.variants.CallsetsSearchResponseCheckIT.ggetCallSetWithValidIDShouldSucceed(CallsetsSearchResponseCheckIT.java:180)
at junitparams.JUnitParamsRunner.runChild(JUnitParamsRunner.java:416)

Another -- "AvroJsonn," "DatasetsSearchhIT," "Constructtor," and more:

Path not implemented

org.ga4gh.ctk.transport.GAWrapperException: Path not implemented
at org.ga4gh.ctk.transport.avrojson.AvroJson.updateTheRespAndLogMessages(AvroJson.java:2281)
at org.ga4gh.ctk.transport.avrojson.AvroJson.doGetResp(AvroJson.java:337)

at org.ga4gh.ctk.transport.avrojson.AvroJsonn.doGetResp(AvroJson.java:317)
at org.ga4gh.ctk.transport.protocols.Client$Reads.getDataset(Client.java:366)
at org.ga4gh.cts.api.datasets.DatasetsSearchhIT.checkSearchResultAgainstGet(DatasetsSearchIT.java:78)
at org.ga4gh.ctk.AntExecutor.executeAntTask(AnntExecutor.java:173)
at org.ga4gh.ctk.TestRunner.doTestRun(TestRunner.java:103)
at org.ga4gh.ctk.TestRunneer.doTestRun(TestRunner.java:70)
at org.ga44gh.ctk.Application.run(Application.java:70)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:7077)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:691)
at org.springframework.boot.SpringAppliccation.afterRefresh(SpringApplication.java:678)
at org.springframework.boot.SpringApplication.doRun(SpringgApplication.java:343)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:273)

at org.springframework.boot.SpringApplication.run(SpringApplicatioon.java:971)
at org.springframework.boot.SpringApplication.run(SpringApplicatioon.java:960)
at org.ga4gh.ctk.Application.main(Application.java:53)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread.run(Thread.jaava:745)
Caused by: org.ga4gh.methods.GAException
at java.lang.reflect.Constructor.newInstance(Constructtor.java:422)

Following instructions on how to run the compliance tests (thanks @hjellinek! ) I'm finding the URL configured in parent/pom.xml is not used when running the tests under maven. It there somewhere else this neede to be confugured too?

Details:

1. parent/pom.xml edited to use: <ctk.tgt.urlRoot>http://localhost:3000/ga4gh

2. mvn clean install

messages use default server URL::

[main] INFO org.ga4gh.ctk.transport.avrojson.AvroJson - create interaction for http://localhost:8000/v0.5.1/datasets/search SearchDatasetsRequest SearchDatasetsResponse

The build appears to be successful

3. mvn test failsafe:integration-test

default URL used again:

[main] INFO org.ga4gh.ctk.transport.avrojson.AvroJson - create interaction for http://localhost:8000/v0.5.1/variantsets/search SearchVariantSetsRequest SearchVariantSetsResponse
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<< FAILURE! - in org.ga4gh.cts.api.variants.VariantsSearchIT

Several of the POM files mention various Jackson modules as dependencies, but I believe this is no longer the case.

Investigate this. If Jackson is no longer used, remove those dependencies.

From: Jerome Kelleher
One immediate problem I had was that the kit automatically appends "v0.5.1" to the URL
provided. URL versioning isn't part of the API. Implementers are free to have whatever URL
prefix they like, but this should be supplied as part of the URL argument. Because of this, I
wasn't able to get much of a sense of what's happening test-wise.

I

This is related to ga4gh/ga4gh-schemas#381

Currently the tests appear to require that IDs be assigned in the following form, which is presumably the behavior of the reference server:
datasetName(or ID?):sampleName(?):readGroupName(?)

We either need to clarify the documentation on ReadGroup.name and/or ReadGroup.id as to how it relates to the read group ID which comes off a sequencer, or we need to describe in the test-data section how it expects read group names to be assigned upon import. Though the docs are especially murky on ReadGroup.id, it should follow the pattern of our other resource IDs and the client should not assume that they have control over the value.

The integration test should use SearchReadGroupSets() to acquire the actual ReadGroupIds for later use in testing SearchReads.

These tests look to be written based on specific implementation details of the reference server, I would not expect them to pass on any other implementation. For example, a GET request to /callsets/search is interpreted by the Google API as a request for a callset with id 'search'.

To me, these looks more like server integration tests which I would expect to live in the reference server repository, if anywhere.

The schemas/API don't mention a "server landing page," so we shouldn't be testing the existence of such a thing.

Which is to say, remove class LandingPageIT.

Automatic code style checks save a lot of time. It would be nice to have an automatic check executed as part of the continuous integration tests.

This would help clients get a sense of how fast their API was in comparison to others and/or previous runs. Obviously it would be a rough metric, but even an overall time could be useful.

@calbach reports that URLs are most likely not being constructed properly - all of the URLs shown in the logs are incorrect, e.g.:

https://www.googleapis.com/genomics/v1beta2?key=AIzaSyDSCPSt4PhkKoVc8NlcscE_uPvlUppVCEY//callsets/search (checkSearchRouting)

Should actually be:

https://www.googleapis.com/genomics/v1beta2/callsets/search?key=AIzaSyDSCPSt4PhkKoVc8NlcscE_uPvlUppVCEY

Issues that I've found:

• ReadGroupSetsSearchIT.readGroupSetsNonexistentNameShouldSayNotFound
  • Should expect an empty list
• ReadGroupSetsGetByNameIT.testSearchForBogusNameFails (this appears to test duplicate functionality at the above test?)
  • Should expect an empty list
• ReadGroupSetsSearchIT.readgroupSetResponseForNonexistentDatasetIdShouldReturnEmptyList()
  • Should expect a 404, since the dataset does not exist. This is not freeform criteria - this is a container object.

Some of these points perhaps should be clarified in the spec, but the compliance test is either being more specific than the spec, or is in conflict with the spec on this point.

Add a test to ReadsPagingIT or ReadGroupSetsPagingIT:

• pick two relatively prime page sizes, A and B
• while returned page token != null, fetch objects using page size A, collect in set SA
• while returned page token != null, fetch objects using page size B, collect in set SB
• assert that SA contains the same elements as SB, and vice-versa.

All of the pom.xml files that mention a version mention "0.5.1-SNAPSHOT". We will need to change that, though I don't know what the new version will be yet.

Main Questions

• 1. For the 5 BAM files in the reads folders, am I right to assume that there should be exactly one readgroupset per file?
• 2. The supplied references don't match the reads -- is that intentional?
• 3. For references, instead of requiring a custom set of references, I suggest instead requiring one of the known (and checksummed) standard referencesets. Any practical implementation will need that anyway. (I don't mind also having custom references in the tests, but it seems much lower priority.)

Minor Data Comments

• 4. How do we decide how much test data it's important to have, and what extra value we get from having more? For example, how much better are the tests because we have these five BAMs instead of just one or two, and these ten VCFs instead of just one or two? (Not a problem to have more, but I want to make sure we're being thoughtful about when to have more files and what each new one adds.)
• 5. It would be nice if it were easy to download just the compliance data, without also getting all of the code (not a big deal, since the code’s not big today, but a bit of a nuisance).
• 6. For import into the Google server, it would be a tiny bit more convenient if the BAMs and VCFs were 'flatter', instead of being spread across multiple folders. If nothing else, can we avoid having duplicate filenames in different directories (like currently happens with the VCFs)?

In the variant tests, info fields are assigned the type keyvalue where they are actually arrays of key-values.

The tests in ReferenceBasesPagingIT should not assume there's a single reference. There may be several.

Find and fix other places where tests make this assumption.

It would be very useful if the compliance tests characterised the correct behavior in some error conditions. For example:

• Invalid URL -> HTTP 404
• Content type != "application/json" -> HTTP 415

What is the boundary between what returns a HTTP error and what returns HTTP success but a GAException object? For example, what happens when the content cannot be parsed as JSON? Is this a HTTP error or a protocol error?

I guess these issues should be documented in the protocol descriptions somewhere, but it might be a good place to start to cover some of the 'easy' cases first in the compliance tests.

Split it into logical groups (classes) that relate to the objects they handle.

This can only happen after the merge of the _tests branches, because it will affect the packages they contain.

Hi,
developing a ga4gh server (https://github.com/med-at-scale/high-health), we hit some issues with compliance tests.

Specifically, in the POST /variantsets/search call, avro JSON deserialization requires a message in the form:

{"datasetIds":["1"], "pageSize": null, "pageToken": null}

i.e. PageSize and pageToken must be present (I admit for fields with default values in the schema it is puzzling), while the compliance suite sends:

{"datasetIds":["1"]}

Any hint? WDYT?
Should we be more lenient with the messages coming in and deal with the 'mismatches'? Do you do that in the python server implementation? Or is the compliance app supposed to send more 'schema compliant' messages (^^)?

Xavier

The difference between assertFields and assertArrayObject is really unclear and easy to get wrong.

It probably makes most sense to just use one method, and pass in a boolean to indicate whether the object being tested is an array or something.

[o.g.c.t.a.AvroJson ] stubbing future comms due to problem communicating JSON with http://localhost:8090/variants/search

I would prefer this behavior were removed entirely - tests should ideally be completely independent of one another, aside from required setup/teardown.

I've observed this happening when the integration test receives an unexpected 400 error. The result is that all subsequent test cases fail as well in an opaque manner (mostly with null pointer exceptions). Often times, the issue has been that the Google API returns a 400 (invalid argument) where the client expected something like a 404 - these cases arise frequently where the spec is unclear.

Example NPE following that message:

java.lang.NullPointerException cannot be cast to org.ga4gh.ctk.transport.GAWrapperException

java.lang.ClassCastException: java.lang.NullPointerException cannot be cast to org.ga4gh.ctk.trransport.GAWrapperException
at org.ga4gh.cts.api.Utils.catchGAWrapperExcepption(Utils.java:316)
at org.ga4gh.cts.api.variants.VariantsMethodsEndpointAliveIT.testSearchVariantsForNonexistentReferenceFails(VariantsMethodsEndpointAliveIT.java:46)
at org.ga4gh.ctk.AntExecutor.executeAntTask(AntExecutor.java:173)
at org.ga4gh.ctk.TestRunner.doTestRun(TestRunner.java:103))
at org.ga4gh.ctk.TestRunner.doTestRun(TestRunner.java:70)
at org.ga4gh.ctk.Application.run(Application.java:70)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:707)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:691)
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:678)
at org.springframework.boot.SpringApplication.doRun(SpringApplication.javaa:343)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:273)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:971)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:960)
at org.ga4gh.ctk.Application.main(Application.javaa:53)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread..run(Thread.java:745)

Add a second variant set (VCF) to the compliance data comprised of "clinically relevant" variant entries from clinvar. Note that these variants have no sample ID database - and thus no reads - backing them.

It will be cool if we can specify the list of endpoints that support GA4GH API
Right now using #cassiedoll 's benchmark I can find only google API working
From the endpoint list we have:

EBI - http://193.62.52.16
Google - https://www.googleapis.com/genomics/v1beta?key=AIzaSyDSCPSt4PhkKoVc8NlcscE_uPvlUppVCEY
Ensembl - http://193.62.52.232:8081
NCBI - http://trace.ncbi.nlm.nih.gov/Traces/gg

I suspect this is a case of overfitting to the reference server.

ReadMethodsEndpointAliveIT

Expecting responseStatus of: <org.ga4gh.ctk.transport.WireTracker@4269e2df> to be: <NOT_IMPLEMENTED> but was: <null>

Per @sarahhunt's suggestion in PR #27, re-architect so that the schemas are freshly loaded from the official GitHub repo every time the compliance suite is built. This would greatly aid in automating a continuous integration workflow between all three repos.

The expected layout of the test data in terms of reference sets, readgroupsets, variant sets etc is not described. There should be a section in the README_COMPLIANCE_DATA.md (which could perhaps be renamed to README.md since it's in the test-data directory?) which describes the expected data layout so server implementers know how they are supposed to import it.

It's perfectly reasonable to say that (e.g.) we require a dataset with name equal to "compliance_dataset_1" must contain exactly one variantset with name "variantset1" containing the variants in the VCF file X. Similar instructions should be given for readgroupsets and referencesets.

Installing JVMs is still not a solved problem. We should probably have a JVM install hints section.

For Mac OS/X, Apple no longer supports java and 1.8 needs to be obtain from Orcale.

I install Maven via MacPorts, but we should also include instructions for Brew.

Maven doesn't pick up the right version of java classes due to JAVA_HOME not being set.
Setting environment variables on OS/X has been rather complex and setting in the .bashrc
with this command is what is suggested:

    export JAVA_HOME=$(/usr/libexec/java_home)

So that wasted an hour and got me to the point of running mvn

Hi all, I didn't want to add noise to the PR thread. I've cloned this branch and I'm unable to build it locally.

compliance(compliance_redux)$ mvn install

[INFO] Scanning for projects...
Downloading: http://repo1.maven.org/maven2/org/springframework/spring-framework-bom/4.2.0.RC2/spring-framework-bom-4.2.0.RC2.pom
[INFO] Unable to find resource 'org.springframework:spring-framework-bom:pom:4.2.0.RC2' in repository central (http://repo1.maven.org/maven2)
[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error building POM (may not be this project's POM).


Project ID: org.springframework:spring-framework-bom

Reason: POM 'org.springframework:spring-framework-bom' not found in repository: Unable to download the artifact from any repository

  org.springframework:spring-framework-bom:pom:4.2.0.RC2

from the specified remote repositories:
  central (http://repo1.maven.org/maven2)

 for project org.springframework:spring-framework-bom

The current data includes but a single Reference. We need multiple of every sort of object to have confidence that our tests, APIs, and server implementations work as expected.

Please add at least two more References to thee test data.

I'm not sure I understand how this is passing for the reference server, if it is.

https://github.com/ga4gh/compliance/blob/compliance_redux/cts-java/src/test/java/org/ga4gh/cts/api/references/ReferencesSearchIT.java#L50
https://github.com/ga4gh/compliance/blob/compliance_redux/cts-java/src/test/java/org/ga4gh/cts/api/references/ReferenceBasesSearchIT.java#L73

The Google Genomics API returns a 400 error, because this is not a valid hexadecimal string.

From @calbach:

... a few issues in the detailed installation section:

• Ideally most of these steps should be eliminated or wrapped up into a helper script. If they cannot be eliminated for the time being, textual descriptions such as "Create a directory target for the test reports to go into.", should be replaced with an actual command line.
• jar xf ctk-cli-0.5.1-SNAPSHOT.jar antRunTests.xml produced nothing for me - I had to copy the version from ctk-server/lib
• chmod +c should probably be chmod +x

When we run a server such that the API commands map to the server root, getUrl prepends an extra / to the path. For example, if we have http://localhost:8000 as the API endpoint, we get URLs like http://localhost:8000//variants/search.

Hi Cassie,

Here's a funny thing:

When I launch the compliance tests via http://cassiedoll.github.io/compliance-staging.html I get 91/165 for Google.

When I git-clone the compliance tests from https://github.com/ga4gh/compliance and launch it with a python SimpleHTTPServer, I get 0/143 for Google.

Shouldn't they behave the same?

Thanks,
Paul

Would it be possible to change the search coordinates for the test cases to something that is closer to the start of chromosome 22 (16057310 for example)? Currently, if we run the compliance tests against the reference server using the example data, we don't get full marks on the variants API because the required variant is in not in the set.

I realise I could just change the example data, but it would be rather a lot easier to change the tests, unless there was some underlying reason for choosing the current coordinates. We do need to have a lightweight example data set, and the easiest way of creating this is to take the heads of the 1kg data. We should be able to pass the compliance tests on small example data sets so that people can develop and test with minimal hardware requirements.

Remove the checked-in code from ctk-domain/src/main/assertj-assertions and ctk-transport/src/main/assertj-assertions and generate it at compile (actually, generate-sources) time.

This will remove ~14,000 lines of code from the repo, with little effect on compilation speed.

Minor issue - we have not implemented the reads API but have 3 reads tests passing as the missing endpoints return 404 errors.

This will have to wait until exception handling is fully agreed and documented, but the tests should distinguish between absence of endpoint and lack of data.

Working my way through the new quickstart, Maven >= 3.2.5 is needed. Debian Jessie has 3.0.5. Do we really need 3.2.5? People like me (i.e., Debian users) really resent having to use newer versions of things than Debian provides!

./ctk --ctk.tgt.urlRoot=https://www.googleapis.com/genomics/v1beta2?key=AIzaSyDSCPSt4PhkKoVc8NlcscE_uPvlUppVCEY --cfg.tgt.dataset_id=3556965148596575732
...
T.TRAFFIC] SearchReadsRequestPOST <{"readGroupIds":["compliance-dataset1:wgBam:wgEncodeUwRepliSeqBg02esG1bAlnRep1_sample","compliance-dataset1:wgBam:wgEncodeUwRepliSeqBg02esG2AlnRep1_sample","compliance-dataset1:1kg-low-coverage:HG00096.mapped.ILLUMINA.bwa.GBR.low_coverage.20120522","compliance-dataset1:1kg-low-coverage:HG00533.mapped.ILLUMINA.bwa.CHS.low_coverage.20120522","compliance-dataset1:1kg-low-coverage:HG00534.mapped.ILLUMINA.bwa.CHS.low_coverage.20120522"]}> null 503
[T.TRAFFIC] SearchVariantSetsRequestPOST <{"datasetId":"compliance-dataset1"}> null 503
[T.TRAFFIC] SearchReadsRequestPOST <{"readGroupIds":[""]}> null 503
[T.TRAFFIC] GET <null> null 503
[T.TRAFFIC] SearchReferencesRequestPOST <{"md5checksums":[],"accessions":[]}> null 503
[T.TRAFFIC] SearchReadsRequestPOST <{"readGroupIds":["myNonsenseId"]}> null 503
[T.TRAFFIC] SearchReadGroupSetsRequestPOST <{"datasetId":"compliance-dataset1","name":"xyzzy"}> null 503
[T.TRAFFIC] SearchReferencesRequestPOST <{"md5checksums":["TODO"],"accessions":[]}> null 503

Requests are still made with 'compliance-dataset1'.

Compliance test should run out-of-the-box with minimal configuration of only server base URL for normal case.

From: Jerome Kelleher
I've run the tests, and it does seem to be quite a powerful framework. However, it's much
too complicated to run at the moment; it took me a good hour to get the directory structure
right (why do we need a lib directory? What do log4j2.xml and antRunTests.xml do?) and get
the tests running. Flexibility is great, but I wonder if things have gone a bit overboard
here. However, this is something we should try to fix after we merge.

The HTML report for org.ga4gh.cts.api.variants.CallsetsSearchResponseCheckIT is reporting checkSearchRouting errors for reads,readgroupsets and references

Are there a few missplaced or unneccessary makeUrl calls in:

cts-java/src/test/java/org/ga4gh/cts/api/variants/CallsetsSearchResponseCheckIT.java

I'm trying again to run this against the Google API. Some of these tests I expect to fail on account of the 'TODO' issue (#54). Others I would guess are due to the change to make referenceSetId required (ga4gh/ga4gh-schemas#392)?

./ctk --ctk.tgt.urlRoot=http://localhost:8090 --ctk.tgt.dataset_id=3556965148596575732 --ctk.matchstr='**/*Reference*'
[TESTLOG] Suite start org.ga4gh.cts.api.references.ReferenceBasesSearchIT
[TESTLOG] ERROR: searchForExpectedReferenceBases(org.ga4gh.cts.api.references.ReferenceBasesSearchIT) due to org.apache.avro.AvroRuntimeException: Field referenceSetId type:STRING pos:0 not set and has no default value
[TESTLOG] ERROR: getAllReferencesWithEmptyMd5List(org.ga4gh.cts.api.references.ReferenceBasesSearchIT) due to org.apache.avro.AvroRuntimeException: Field referenceSetId type:STRING pos:0 not set and has no default value
[TESTLOG] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.215 sec
[o.g.c.Application ] task [junit] tgt [tests] msg: Test org.ga4gh.cts.api.references.ReferenceBasesSearchIT FAILED
[TESTLOG] task [junit] tgt [tests] msg: Test org.ga4gh.cts.api.references.ReferenceBasesSearchIT FAILED
[TESTLOG] Suite start org.ga4gh.cts.api.references.ReferenceSetsSearchIT
[TESTLOG] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.968 sec
[TESTLOG] Suite start org.ga4gh.cts.api.references.ReferencesSearchIT
[TESTLOG] ERROR: searchForExpectedReferences(org.ga4gh.cts.api.references.ReferencesSearchIT) due to org.apache.avro.AvroRuntimeException: Field referenceSetId type:STRING pos:0 not set and has no default value
[TESTLOG] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.005 sec
[o.g.c.Application ] task [junit] tgt [tests] msg: Test org.ga4gh.cts.api.references.ReferencesSearchIT FAILED
[TESTLOG] task [junit] tgt [tests] msg: Test org.ga4gh.cts.api.references.ReferencesSearchIT FAILED
[TESTLOG] Suite start org.ga4gh.cts.api.references.ReferencesTestSuite
[TESTLOG] ERROR: initializationError(org.ga4gh.cts.api.references.ReferencesTestSuite) due to URI is not hierarchical
[TESTLOG] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 sec
[o.g.c.Application ] task [junit] tgt [tests] msg: Test org.ga4gh.cts.api.references.ReferencesTestSuite FAILED
[TESTLOG] task [junit] tgt [tests] msg: Test org.ga4gh.cts.api.references.ReferencesTestSuite FAILED
[TESTLOG] Suite start org.ga4gh.cts.api.references.ReferencesTests
[TESTLOG] ERROR: initializationError(org.ga4gh.cts.api.references.ReferencesTests) due to Test class should have exactly one public constructor
[TESTLOG] ERROR: initializationError(org.ga4gh.cts.api.references.ReferencesTests) due to No runnable methods
[TESTLOG] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.004 sec
[o.g.c.Application ] task [junit] tgt [tests] msg: Test org.ga4gh.cts.api.references.ReferencesTests FAILED
[TESTLOG] task [junit] tgt [tests] msg: Test org.ga4gh.cts.api.references.ReferencesTests FAILED
[TESTLOG] Overall: Tests run: 7, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 1.196 sec

In the compliance_redux branch, make the ID of the test dataset a parameter that can get passed in from the command line.

Hi @hjellinek !

I've picked up the latest version of the compliance suite after the recent flurry of activity and run it against my dev server which has post v0.5.1 features.

The checkComplianceDatasetIsPresent test in DatasetsSearchIT.java is failing for me because it expects the compliance dataset to be the first returned, when we currently return it as the third record. It would be better to loop through the returned datasets than assume ordering.

On line 24 the hardcoded conversion points to a file that's not there.

david4096@f4191ec

Change the logging level of the

org.ga4gh.ctk.transport.avrojson.AvroJson - set urlRoot = http://localhost:8000 path = ...

message from INFO to DEBUG.

From: Jerome Kelleher
I agree with @calbach that it would be very helpful for the HTML test reports to contain the
actual messages transferred. This was a genuinely useful part of the old compliance tests.

This appears not to be the case:
ga4gh/ga4gh-server#758

"compliance-dataset1" is tricky if you want to parse the version out of it (e.g. sorting for finding the newest), suggest ga4ga-compliance-dataset-1 or something else that is easy to just split rather
than a regexp.

We should add Maven Enforcer rules to set the minimum Java and Maven versions we support.

Java = JDK 1.8.
Maven = 3.3.3 (apparently)

There's a bug in most places where the compliance tests expect HTTP errors, as the convention is to blindly dereference a caught exception (may be null, if the server returned a 200), thereby potentially causing an irrelevant stack trace.

For example, ReadGroupSetsGetByNameIT.testSearchForBogusNameeFails incorrectly expects the server to a return a 404 if it doesn't know about the bogus name (in actuality, it should expect an empty list of readGroupSets, see ga4gh/ga4gh-schemas#382). I'll file a separate issue for that, but this was difficult to debug because the test code actually attempts to dereference a null:

testSearchForBogusNameFails Error   N/A

java.lang.NullPointerException
at org.ga4gh.cts.api.reads.ReadGroupSetsGetByNameIT.testSearchForBogusNameeFails(ReadGroupSetsGetByNameIT.java:50)
at org.ga4gh.ctk.AntExecutor.executeAntTask(AntExecutor.java:173)
at org.ga4gh.ctk.TestRunner.doTestRun(TestRunner.java:101))
at org.ga4gh.ctk.TestRunner.doTestRun(TestRunner.java:70)
at org.ga4gh.ctk.Application.run(Application.java:70)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:707)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:691)
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:678)
at org.springframework.boot.SpringApplication.doRun(SpringApplication.java:343)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:273)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:971)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:960)
at org.ga4gh.ctk.Application.main(Application.java:53)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread..run(Thread.java:745)

Ideally, the test would instead print an error which explains what the test expects, and what was actually returned.