edgarrmondragon / tap-clinicaltrials

Singer tap for ClinicalTrials.gov study records data.

tap-clinicaltrials

Singer tap for ClinicalTrials.gov study records data.

Built with the Meltano Tap SDK for Singer Taps.

Capabilities

• catalog
• state
• discover
• about
• stream-maps

Settings

Setting	Required	Default	Description
start_date	False	None	Earliest datetime to get data from
condition	False	None	Conditions or disease query
sponsor	False	None	Sponsor query
stream_maps	False	None	Config object for stream maps capability. For more information check out Stream Maps.
stream_map_config	False	None	User-defined config values to be used within map expressions.
flattening_enabled	False	None	'True' to enable schema flattening and automatically expand nested properties.
flattening_max_depth	False	None	The max depth to flatten schemas.
batch_config	False	None

A full list of supported settings and capabilities is available by running: tap-clinicaltrials --about

Installation

In a Meltano project

Using a direct reference

meltano add extractor tap-clinicaltrials --from-ref=https://raw.githubusercontent.com/edgarrmondragon/tap-clinicaltrials/main/plugin.yaml

Requires Meltano v3.1.0+.

From MeltanoHub

Not yet available.

From PyPI

python3 -m pip install --upgrade tap-clinicaltrials

With pipx

pipx install tap-clinicaltrials

From source

git clone https://github.com/edgarrmondragon/tap-clinicaltrials
cd tap-clinicaltrials
python3 -m pip install .

Usage

You can easily run tap-clinicaltrials by itself or in a pipeline using Meltano.

With Meltano

1. Clone the repo and cd into it:

   git clone https://github.com/edgarrmondragon/tap-clinicaltrials.git
   cd tap-clinicaltrials

2. Make sure you have Meltano installed

3. Install all plugins

   meltano install

4. Configure the tap-clinicaltrials tap:

   meltano config tap-clinicaltrials set start_date '2020-01-01'
   meltano config tap-clinicaltrials set condition 'COVID-19'
   meltano config tap-clinicaltrials set sponsor 'Pfizer'

5. Run a test tap-clinicaltrials extraction

   meltano run tap-clinicaltrials target-duckdb

6. That's it! Check the data

   $ duckdb output/warehouse.duckdb -c "select nctid, lastUpdateSubmitDate, protocolsection->>'$.identificationModule.briefTitle' from clinicaltrials.studies limit 5;
   ┌─────────────┬──────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────┐
   │    nctid    │ lastupdatesubmitdate │                      (protocolsection ->> '$.identificationModule.briefTitle')                      │
   │   varchar   │       varchar        │                                               varchar                                               │
   ├─────────────┼──────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ NCT06156215 │ 2023-12-06           │ PROmotion of COVID-19 BOOSTer VA(X)Ccination in the Emergency Department - PROBOOSTVAXED            │
   │ NCT05487040 │ 2023-12-06           │ A Study to Measure the Amount of Study Medicine in Blood in Adult Participants With COVID-19 and …  │
   │ NCT06163677 │ 2023-12-07           │ A Study to Look at the Health Outcomes of Patients With COVID-19 and Influenza.                     │
   │ NCT05032976 │ 2023-12-07           │ Korea Comirnaty Post-marketing Surveillance                                                         │
   │ NCT05596734 │ 2023-12-11           │ A Study to Evaluate the Safety, Tolerability, and Immunogenicity of Combined Modified RNA Vaccine…  │
   └─────────────┴──────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘

Executing the Tap Directly

tap-clinicaltrials --version
tap-clinicaltrials --help
tap-clinicaltrials --config CONFIG --discover > ./catalog.json

Developer Resources

Initialize your Development Environment

pipx install hatch

Create and Run Tests

Run integration tests:

hatch run test:integration

You can also test the tap-clinicaltrials CLI interface directly:

hatch run sync:console -- --about --format=json

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.

tap-clinicaltrials's Issues

Support native field selection

The API supports querying specific fields with the fields query parameter, which is a comma-separated list of fully-qualified field names, e.g. protocolSection.identificationModule.nctId.

Problem is this might result in URL that's too long (status 414).