ga4gh / drs-compliance-suite

Tests to verify the compliance of a DRS implementation with GA4GH DRS specification

drs-compliance-suite

Tests to verify the compliance of a DRS implementation with GA4GH Data Repository Service (DRS) specification. This compliance suite currently supports the following DRS versions and will aim to support more versions of DRS in the future.

• DRS 1.2.0

Installations

• Python 3.x is required to run DRS Compliance Suite natively or using PyPI package.
• Docker Desktop is required to run DRS Compliance Suite using a docker image.

Running DRS Compliance Suite

1. Natively

Install the packages from requirements.txt

cd drs-compliance-suite
pip3 install -r requirements.txt

Add PYTHONPATH to env variables

export PYTHONPATH=<absolute path to drs-compliance-suite>

Run the compliance suite

python3 compliance_suite/report_runner.py --server_base_url "http://localhost:8085/ga4gh/drs/v1" --platform_name "ga4gh starter kit drs" --platform_description "GA4GH reference implementation of DRS specification" --drs_version "1.2.0" --config_file "compliance_suite/config/config_samples/config_basic.json" --serve --serve_port 56565

Note: This specific command is an example of running the compliance suite on a local deployment of DRS that is running on port 8085.
When running the compliance suite, it's important to configure the command line arguments according to the specific DRS implementation you're testing. Please refer to the Command Line Arguments section for details on each of these arguments.

2. Using PyPI Package

Install the latest version of the drs-compliance-suite PyPI package using pip3

pip3 install drs-compliance-suite --upgrade

Run the compliance suite

drs-compliance-suite --server_base_url "http://localhost:8085/ga4gh/drs/v1" --platform_name "ga4gh starter kit drs" --platform_description "GA4GH reference implementation of DRS specification" --drs_version "1.2.0" --config_file "compliance_suite/config/config_samples/config_basic.json" --serve --serve_port 56565

Note: This specific command is an example of running the compliance suite on a local deployment of DRS that is running on port 8085.
When running the compliance suite, it's important to configure the command line arguments according to the specific DRS implementation you're testing. Please refer to the Command Line Arguments section for details on each of these arguments.

3. Using Docker

Pull the latest docker image from dockerhub.

docker pull ga4gh/drs-compliance-suite:1.0.3

Run the compliance suite using the docker image. Make sure your config file is available at config/config_drs.json

docker run -d --name drs-compliance-suite -v $(PWD)/output/:/usr/src/app/output/ -v ${PWD}/config/:/usr/src/app/config/ -p 57568:57568 ga4gh/drs-compliance-suite:1.0.3 --server_base_url "http://host.docker.internal:8085/ga4gh/drs/v1" --platform_name "ga4gh starter kit drs" --platform_description "GA4GH reference implementation of DRS specification" --report_path "./output/drs-cs-report.json" --drs_version "1.2.0" --config_file "./config/config_drs.json" --serve --serve_port 57568

Note:

• To run this docker image on MAC ARM processor, append --platform linux/x86_64 to the above command.
• This specific command is an example of running the compliance suite on a local deployment of DRS that is running on port 8085.
  When running the compliance suite, it's important to configure the command line arguments according to the specific DRS implementation you're testing. Please refer to the Command Line Arguments section for details on each of these arguments.

Command Line Arguments

Command Line Argument	Description	Optional/Required	Default Value
--server_base_url	The base URL of the DRS implementation that is being tested by the compliance suite.	Required	N/A
--platform_name	The name of the platform hosting the DRS server.	Required	N/A
--platform_description	The description of the platform hosting the DRS server.	Required	N/A
--drs_version	The version of DRS implemented by the DRS server taht is being tested for compliance. It can be one of the following: "1.2.0"	Required	N/A
--config_file	The file path of the JSON config file. The config file must contain auth information for service-info endpoint and different DRS objects. Refer to the config-file section for more details.	Required	N/A
--report_path	The path of the output JSON report file.	Optional	"./output/drs_compliance_report.json"
--serve	If this flag is set to True, the output report is served as an HTML webpage at the port specified by --serve_port.	Optional	False
--serve-port	The port where the output report HTML is deployed.	Optional	57568

Config File:

The compliance suite is provided with information for testing the DRS server through a user-provided JSON config file. This file includes the following details:

• Authorization information for the service-info endpoint
• A few DRS Object IDs that are present in the DRS server
• Authorization information for each of these DRS objects
• Indication of whether the DRS object is a bundle or a single blob

Here's a template for a config file that can be used to configure these details:

{
  "service_info": {
      "auth_type": "basic",
      "auth_token": "dXNlcm5hbWU6cGFzc3dvcmQ="
  },
  "drs_objects" : [
      {
          "drs_id": "697907bf-d5bd-433e-aac2-1747f1faf366",
          "auth_type": "none",
          "auth_token": "",
          "is_bundle": false
      },
      {
          "drs_id": "0bb9d297-2710-48f6-ab4d-80d5eb0c9eaa",
          "auth_type": "basic",
          "auth_token": "dXNlcm5hbWU6cGFzc3dvcmQ=",
          "is_bundle": false
      },
      {
          "drs_id" : "41898242-62a9-4129-9a2c-5a4e8f5f0afb",
          "auth_type": "bearer",
          "auth_token": "secret-bearer-token-1",
          "is_bundle": true
      },
      {
          "drs_id" : "a1dd4ae2-8d26-43b0-a199-342b64c7dff6",
          "auth_type": "passport",
          "auth_token": "43b-passport-a1d",
          "is_bundle": true
      }
  ]
}

• The "auth_type" specifies the type of authorization, which can be one of the following: ["basic", "bearer", "passport", "none"]
• The "auth_token" field contains the corresponding authorization token value.
• If "auth_type" is set to "basic", the "auth_token" can be created by using base64 encoding.
• If "auth_type" is set to "none", the "auth_token" field should be left blank with a value of "".
• The "is_bundle" flag indicates whether the DRS object is a bundle or a single blob. If set to True, the object is a bundle, and if set to False, the object is a single blob.

You can find some sample config files here

Unittesting

Run the unittests with coverage

pytest --cov=compliance_suite unittests/

Changelog

v1.0.3

• provide flexibility in providing different auth information for drs object and drs access endpoints
• remove incorrect skip status setting

v1.0.2

• Reduce the docker image size by using python:3.11-slim-bullseye instead of python:3

v1.0.1

• Fixed a bug in the docker deployment of DRS Compliance Suite
• Update README documentation

v1.0.0

• DRS Compliance Suite for Data Repository Service v1.2.0

drs-compliance-suite's Issues

Add CWL workflow for drs-compliance-suite

Add WDL workflow for drs-compliance-suite

set status of a test case to "warn" when there are extra properties in the response schema

Add unittests to test the service-info, get object tests and report generation for good_mock server

Implement compliance tests for all endpoints

Create drs mock servers for unittesting

A drs server that works as per DRS v1.2 spec (good mock server) and another DRS that doesn't conform to DRS spec (bad mock server). These server endpoints will be used to test the compliance suite.

Implement bundles, expand parameter in good_mock_server

Add unittests to test the service-info, get object tests and report generation for bad_mock server

Add to report: information about what type of auth was used for tests

It would be useful if the reporting of the tests included what type of auth was used for each test.

Note: the specific tokens should not be listed.

Add bearer token auth to mock servers and compliance_suite

Add basic auth to mock servers and compliance_suite

Add an input argument that provides details on any authentication (basic/bearer/passport) that is implemented in the DRS server.
Modify the mock servers and compliance tests to handle the same.

Support implementations that do not require authorization for the get_object endpoint

For objects that require authorization...

Many DRS implementations do not require authorization for get_object. Authorization is only needed when obtained the url at which to download bytes.

This applies to all the gen3 DRS implementations (crdc, topmed, etc.) and to the NCBI implementation.

The compliance suite appears to apply authentication to the get_objects request i.e. the get /objects/ end point.

The first result set (drs_compliance_report2.txt) shows the test report for the NCBI DRS with "auth_type": "passport"
This fails the "drs object info" phase because authentication is being applied to the request. The server does not expect auth to be passed for this request.

The second result run (drs_compliance_report2.txt) used the same drs ids with with "auth_type": "none"
The responses for the ids pass most tests
(the issue with the NCBI implementation providing self_uri as self_url is a separate issue and will be dealt with by the implementation)
It of course fails the "drs object access" phase with a 401 when it attempts to get the url for the object.

The compliance suite should support applying auth only to the object access phase in such a scenario.

Note that some implementations require auth for both phases. The compliance suite will need to handle both scenarios.

The attached .txt files are json, but the json extension was not supported during file upload to GitHub.

drs_compliance_report2.txt
drs_compliance_report3.txt

Generate output json report that follows the testbed-lib schema

Add test cases for service-info endpoint

• success case
  • check status code
  • check content type
  • check if all the required fields are present in the service_info response

DRS 1.2 spec does not include version number

The Swagger version of the DRS spec does not list 'version' as a requirement.
https://editor.swagger.io/?url=https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.2.0/openapi.yaml

This is a possible issue because the compliance suite apparently relies on version to determine which checks to perform.

Validate using Python jsonschema & Add test cases for object endpoint

• Add json schemas for all endpoint responses
• update report and report_runner to validate responses using jsonschema
• GET object by object_id
• OPTIONS object by object_id
• POST bulk request objects by object_id
• OPTIONS bulk request objects by object_id

Add github actions

• run unittests
• create and publish docker image
• publish updated pypi package (only on master branch)