pouyan9675 / tri-al

A visualization and analytics platform for clinicaltrials.gov data that enables researcher to interpret the data in a convenient way.

Tri-AL: VisuAL ClinicAL TriALs

Clinicaltrials.gov is a website that people all around the world submit the progress and the information of the medical trials on it. Tir-AL is a visualization platform to keep track of clinicaltrials.gov data along with more valuable tools. Tri-AL allows clinicaltrials researchers explore the database in a novel way. It can be used as both standalone project and a baseline for further improvement. It is still under development and we are trying to make it as perfect as possible.

Setup

1. Activate environment and install all requirements using pip command and requirements.txt file.

pip install -r requirements.txt

2. Intialize database and setting up project.

chmod +x initdb.sh
./initdb.sh

3. Download the concent of clinicaltrials.gov from here as a zip file to initialize database.

4. Unzip the AllPublicXML.zip and place the content under data directory in the project root. For example data/AllPublicXML/.

5. Import all the XML files to the database.

python3 data_manager.py import -i data/AllPublicXML

6. Update config file by using sample cofing file. The config file is located in visual/sample-config.yml. Name config to config.yml in the same directoy. It can be done as following as well:

cp visual/sample-config.yml visual/config.yml
# EDIT config.yml WITH ANY EDITOR

7. Start project using Django and access it through browser!

python3 manage.py runserver

Project Structure

.
├── panels                  # Internal app for tri-al project that contains core of the server
│   ├── api                 # Files that are responsible for REST-API to communicate with front-end
│   └── utils               # Scripts to download and parse clinicaltrials.gov data and insert them into database
├── visual                  # Configuration files for Django app
├── schedulers              # bash scripts for scheduling tasks such as, updating database
├── initdb.sh               # Script to reset the database and make a new one with default values
├── httpd.conf              # Backup file of apache configuration for app
├── data_manager.py         # Main script to maintain database and download and fill database
├── manage.py               # Django script to perform different actions for app
└── .gitignore

tri-al's Issues

nan values for 'study_duration'

nan values for 'study_duration'

Using the data from Clinicaltrial.gov's resources here (e.g. https://classic.clinicaltrials.gov/AllPublicXML.zip).

It looks like the system is getting NaN for all the study_duration values.

Tried shortcircuiting line 297 in panels/utils/process.py if `row['StudyDuration'] is None or isnan(row['StudyDuration'][), but it looks like this is popping up for almost all the studies, so there may be an issue with parsing?

Reproduction

• Follow readme setup instructions
• Run python3 data_manager.py import -i data/AllPublicXML/
• Find the following error trace

0%| | 0/459487 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/fields/init.py", line 1823, in get_prep_value
return int(value)
ValueError: cannot convert float NaN to integer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/.../Tri-AL/data_manager.py", line 168, in <module>
_import(args.input)
File "/.../Tri-AL/data_manager.py", line 122, in _import
t = processor.data_mapper(row.to_dict(orient='index')[0])
File "/.../Tri-AL/panels/utils/processor.py", line 320, in data_mapper
t.save()
File "/.../Tri-AL/panels/models.py", line 249, in save
super(Trial, self).save(*args, **kwargs)
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/base.py", line 739, in save
self.save_base(using=using, force_insert=force_insert,
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base
updated = self._save_table(
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/base.py", line 881, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/base.py", line 919, in _do_insert
return manager._insert(
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1415, in execute_sql
for sql, params in self.as_sql():
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1358, in as_sql
value_rows = [
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
[self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
[self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1300, in prepare_value
value = field.get_db_prep_save(value, connection=self.connection)
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/fields/init.py", line 842, in get_db_prep_save
return self.get_db_prep_value(value, connection=connection, prepared=False)
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/fields/init.py", line 837, in get_db_prep_value
value = self.get_prep_value(value)
File "/.../Tri-AL/venv/lib/python3.10/site-packages/django/db/models/fields/init.py", line 1825, in get_prep_value
raise e.class(
ValueError: Field 'study_duration' expected a number but got nan.

Separately, love the idea behind this project!