spiros / tofu

Tofu is a Python tool for generating synthetic UK Biobank data.

Tofu

Tofu is a Python library for generating synthetic UK Biobank data.

The UK Biobank is a large open-access prospective research cohort study of 500,000 middle aged participants recruited in England, Scotland and Wales. The study has collected and continues to collect extensive phenotypic and genotypic detail about its participants, including data from questionnaires, physical measures, sample assays, accelerometry, multimodal imaging, genome-wide genotyping and longitudinal follow-up for a wide range of health-related outcomes.

Tofu will generate synthetic data which conform to the structure of the baseline data UK Biobank sends researchers by generating random values:

• For categorical variables (single or multiple choices), a random value will be picked from the UK Biobank data dictionary for that field.
• For continous variables, a random value will be generated based on the distribution of values reported for that field on the UK Biobank showcase.
• For date and date/time fields, a random date will be generated.
• For all other fields, such as polymorphic fields, no data will be generated.

Some general observations:

• The lookups directory contains lookups downloaded from the UK Biobank showcase - they might need to be updated when new fields become available.
• Data conform to the structure and schema of the baseline file but are otherwise nonsensical: no checks have been implemented across fields.
• All eid's (patient identifiers) generated from this tool are prefaced with 'fake' in order to avoid confusion with legitimate datasets.
• Dates randomly generated are between 1910 and 1990 again to avoid confusion with real data.

You can find more information on the UK Biobank here:

• The protocol paper in PLOS Medicine
• Presentations from the UK Biobank Scientific Conference contain a lot of information on various aspects of the dataset.
• The UK Biobank Showcase contains information for each field.
• The data dictionaries contain machine-readable metadata for each field.

Usage

Generate synthetic data for 100 patients across all baseline fields (not advised unless you really have the entire dataset):

python tofu.py -n 100
Wrote synthetic-20200128181342.csv shape (100,21946).

Generate synthetic data for 100 patients for fields 3 and 20002

Note: you do not have to specify the eid in the list of fields since all generated datasets include it by default.

python tofu.py -n 100 --field 3 20002
Wrote synthetic-20200128171143.csv shape (10,103).

Generate synthetic data for 100 patients for fields 3 and 20002 and make 10% of values missing.

python tofu.py -n 100 --field 3 20002 -j 10
Wrote synthetic-20200128191124.csv shape (100,103).

Generate synthetic data for 100 patients for fields 3 and 20002 and write decoded fields and values (human readable format).

python tofu.py -n 100 --field 3 20002 -H

Contributing

General points

• Pull requests are welcome! :)
• For major changes, please open an issue first to discuss what you would like to change.
• Please make sure to update tests as appropriate.

Contributors

Many thanks to all who have contributed to this tool:

• Isuru Liyanage for implementing the -H flag for humans
• Alessia Peviani for implementing the -F flag

Cite as

Spiros Denaxas. (2020). spiros/tofu: Updated release for DOI (Version v1.1). Zenodo. http://doi.org/10.5281/zenodo.3634604

License

MIT

tofu's Issues

Updating dictionaries

Hi,
I tried to browse UKB showcase and find updated dictionaries (lookups). But I could only find updated dictionary for all fields (df_lkp_fields.tsv). So I am missing updated values for encoding and stats. Is it possible to update the readme file with links to these files.
Thanks

pd.np => np

tofu/helpers.py

Line 138 in 3525018

pd.np.random.randint(d, size=n),

Add support for loading fields from file

I need to run tofu with specific subsets of fields to generate different synthetic test files, each with variables from a specific domain.
Passing the fields via command line is not always ideal if it's more than just a few, and it would be nice to capture how I generated the output in the first place, especially if I want to re-use these fields in a future run.
I think the solution is to enable a file option, preferably mutually exclusive with the current -f argument.

The file format I propose is the following:

• one field per line
• anything after the field on the same line will be ignored (for example, comments)
• empty lines and lines starting with # will be ignored

# my file - this line is a comment
51
32
# the empty line below is ignored

34 200 # the second number and this comment are ignored

Issues with the missing data flag -j option

Hi Spiros,

The -j flag does not seem to work without specifying specific fields and in some cases does not work depending on the input numbers used. See issue below

python tofu.py -n 100 -j 5
  0%|                                                                                                               | 7/7450 [00:00<04:28, 27.68it/s]
Traceback (most recent call last):
  File "tofu.py", line 96, in <module>
    args.jitter)
  File "/home/johan/projects/tofu/helpers.py", line 238, in insert_missingness
    a[i] = np.nan
  File "/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py", line 4260, in __setitem__
    raise TypeError("Index does not support mutable operations")
TypeError: Index does not support mutable operations

Oddly enough this combination work

python tofu.py -n 20 -j 1

but not this one

python tofu.py -n 100 -j 1
Best Johan and Kenan

Generating decoded values

Thanks for this great tool.
We are working on CINECA project and greatly benefit from this. For the project we are also interested in directly generating decoded values. I have made a fork and created a branch for this feature. If you think this would benefit others, I can create a pull request. Let me know your thoughts or need more information.