bd2kgenomics / protect

ProTECT

Prediction of T-Cell Epitopes for Cancer Therapy

This repo contains the Python libraries for the Precision Immunology Pipeline developed at UCSC.

src/protect/pipeline/ProTECT.py             - The python script for running the pipeline.
src/protect/pipeline/input_parameters.yaml  - The config file for the run that contains all the
                                              necessary parameters for the run.
Flowchart.txt                               - A (super cool) flowchart describing the flow of
                                              the pipeline.

ProTECT uses sequencing information from a patient to predict the neo-epitopes produced in their tumors that can be used in T-cell based, or peptide vaccine based therapies.

All docker images used in this pipeline are available at

                    https://hub.docker.com/u/aarjunrao/

To learn how the pipeline can be run on a sample, head over to the ProTECT Manual

ProTECT is currently in its infancy and is under continuous development. We would appreciate users sharing the level 3 data produced by ProTECT with us such that we can better train our predictive models.

protect's Issues

Protect doesn't see config file

commit dd9b4c7661de6e11e5af7dd0ea0d5304cc60b33e

ProTECT --config_file my_parameters.yaml /mnt/data
gets

ParameterError: The config file was not found at specified location. Please verify and retry.

Also, for some reason the error is written to STDOUT instead of STDERR.

full output at http://hgwdev.cse.ucsc.edu/~jeltje/toil/protect.log

Implement Indel support

Add strelka support

Child of #3 and #2

implement make-config flag

Get a test set for the AACR workshop

I believe this bioproject contains freely available fastqs from a breast cancer patient along with Lymphoblastoid cell lines from the same patient (T/N) This might be helpful

http://www.ncbi.nlm.nih.gov/bioproject/PRJEB2210

This is the matched RNA. Not sure if it's unprotected or not
http://trace.ncbi.nlm.nih.gov/Traces/sra/?run=SRR925710

Use BWA to add read groups

We should use the -R flag to add read groups.

Implement panel of SNV callers

Implement changes to README and MANUAL

update README

Post 2.0

Fix "No Locks Available" Eror

reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:  ---TOIL WORKER OUTPUT LOG---
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:  WARNING:toil.resource:Can't find resource for leader path '/neoantigen_prediction/protect/src/protect'
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/neoantigen_prediction/protect/src', name='protect.pipeline.ProTECT')
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:  CRITICAL:toil.job:CACHE: Unable to acquire lock on /mnt/isilon/maris_lab/target_nbl_ngs/Eric/neoantigen_prediction/test_outputs/toil-64121748-44a6-4d46-bbb4-b0216cfdcd85/cache-64121748-44a6-4d46-bbb4-b0216cfdcd85/.cacheLock
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:  Traceback (most recent call last):
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/worker.py", line 334, in main
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:      with fileStore.open(job):
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:    File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:      return self.gen.next()
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1054, in open
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,614 WARNING: toil.leader: C/w/jobE1qCG6:      with self._CacheState.open(self) as cacheInfo:
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:    File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:      return self.gen.next()
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1826, in open
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:      with outer.cacheLock():
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:    File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:      return self.gen.next()
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1433, in cacheLock
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:      flock(cacheLockFile, LOCK_EX)
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:  IOError: [Errno 37] No locks available
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:  Exiting the worker because of a failed jobWrapper on host reslnmarisig01.research.chop.edu
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host reslnmarisig01.research.chop.edu
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: C/w/jobE1qCG6:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job C/w/jobE1qCG6 to 0
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,615 WARNING: toil.leader: Job: C/w/jobE1qCG6 is completely failed
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:45,640 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: n/i/jobTLYT2d

@hannes-ucsc any comments?

KeyError: 'dbsnp_tbi'

I'm getting a bunch of errors that all contain

File "/home/ubuntu/arjun/protect/src/protect/mutation_calling/muse.py", line 145, in run_muse_sump_perchrom
WARNING: toil.leader: q/B/job6dN4kj:     'dbsnp_coding.vcf.gz.tbi.tmp': muse_options['dbsnp_tbi']}
WARNING: toil.leader: q/B/job6dN4kj: KeyError: 'dbsnp_tbi'

http://hgwdev.cse.ucsc.edu/~jeltje/toil/tbi.err

Add Muse support

child of #3

Rename files in git

Dynamically update disk

Remove Toil from requirements

Add it as a check instead

Add s3am and toil installs to jenkins.sh

No space left on device

ProTECT exits with

IOError: [Errno 28] No space left on device
ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host jeltje-tb

However, my filesystem is fine (jobstore and output are supposed to go to /mnt/data):

Filesystem      Size  Used Avail Use% Mounted on
udev             24G     0   24G   0% /dev
tmpfs           4.8G  247M  4.6G   6% /run
/dev/vda1        49G   19G   31G  38% /
tmpfs            24G     0   24G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            24G     0   24G   0% /sys/fs/cgroup
/dev/vdb1       2.0T  1.3T  554G  71% /mnt
tmpfs           4.8G     0  4.8G   0% /run/user/1000

There is a suspicious Unable to acquire lock on /tmp/toil-0fb5cb74-48c9-408d-b0e5-5c3cf37cce33/cache-0fb5cb74-48c9-408d-b0e5-5c3cf37cce33/.cacheLock so I'm wondering if this is docker being greedy?
Full log at http://hgwdev.cse.ucsc.edu/~jeltje/toil/protect.nospace

Add input_parameters.yaml to the distribution

ProTECT --generate_config fails with the error

Traceback (most recent call last):
  File "/Users/arjun/protect/bin/ProTECT", line 11, in <module>
    sys.exit(main())
  File "/Users/arjun/protect/lib/python2.7/site-packages/protect/pipeline/ProTECT.py", line 565, in main
    generate_config_file()
  File "/Users/arjun/protect/lib/python2.7/site-packages/protect/pipeline/ProTECT.py", line 537, in generate_config_file
    os.path.join(os.getcwd(), 'ProTECT_config.yaml'))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 119, in copy
    copyfile(src, dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/Users/arjun/protect/lib/python2.7/site-packages/protect/pipeline/input_parameters.yaml'```

Use s3am for downloads

Download indexes in separate jobs

Reduces the time taken to download indexes by downloading them all in parallel.

What 'DNA' input does Protect want?

I just noticed the documentation is not specific about WGS or WXS. Is it either?

Support for MAFs

Update s3am description in manual

Make protect with different flavors

Add scripts for MHC pathway gene presence

Make prepare_samples accept S3 urls

typecast dynamic promises to int

related to DataBiosphere/toil/issues/1063

Create mini-references for CI

Use only chr6 so we have ~20k genes, and the MHC

Implement CI testing

use CI input created by @jpfeil and @arkal

Test against pVAC

https://genomemedicine.biomedcentral.com/articles/10.1186/s13073-016-0264-5

implement a fancier SNV consensus caller

star and gencode in generated config are wrong

remake all the dockerfiles and put them in cgl-docker-lib

Remove PAR Y genes form the gtfs and remake all indexes

The pseudo-autosomal-region (PAR) on chrY is the region that is shared between chrX and chrY (this is why those chromsomes pair up during meiosis). The genes on these regions are identical. And just like you wouldn't use two copies of chr1 genes in analysis, you shouldn't use the PAR genes on chrY. They are all named 'ENSTR'

Add setup.py

Make parameter config for CI

Add vardict support

child of #3

'too few arguments' error from ProTECT

Using the make develop method on commit f31ea0a7d1a0f8fdf5ca51664a14d213c113186b
ProTECT --config_file my_parameters.yaml
gets the usage statement, then
ProTECT: error: too few arguments

pip list

bd2k-python-lib (1.13.dev21)
boto (2.38.0)
cgcloud-lib (1.4a1.dev195)
dill (0.2.5)
pip (8.1.2)
protect (2.0, /home/ubuntu/arjun/protect/src)
PyYAML (3.11)
setuptools (12.2)
toil (3.2.0)

pip install doesn't create ProTECT.py

Following instructions at https://github.com/BD2KGenomics/protect/blob/master/MANUAL.md
source ~/venvs/protect/bin/activate
pip list

bd2k-python-lib (1.13.dev21)
boto (2.38.0)
cgcloud-lib (1.4a1.dev195)
dill (0.2.5)
pip (8.1.2)
protect (2.0)   <--- it exists!
PyYAML (3.11)
setuptools (12.2)
toil (3.2.0a2.dev198)

However, ProTECT.py is not in my path:

>which pip
/home/ubuntu/venvs/protect/bin/pip
>which toil
/home/ubuntu/venvs/protect/bin/toil
> which protect
>which ProTECT.py
>

I also cannot autocomplete pro or Pro to a tool, so nothing useful seems to be in my path.
find /home/ubuntu/venvs/protect -iname "protect*"

/home/ubuntu/venvs/protect
/home/ubuntu/venvs/protect/lib/python2.7/site-packages/protect
/home/ubuntu/venvs/protect/lib/python2.7/site-packages/protect-2.0-py2.7.egg-info

Add somatic sniper support

child of #3

Add script for CAR validity

pip install protect gets yaml warnings

pip install protect results in a bunch of warnings like these:

    In file included from ext/_yaml.h:2:0,
                     from ext/_yaml.c:343:
    /usr/include/yaml.h:633:1: note: expected ‘yaml_char_t *’ but argument is of type ‘char *’
     yaml_mapping_start_event_initialize(yaml_event_t *event,

full log here
Everything installs fine, but it clutters the screen. As a new user of any package, these kinds of make warnings always make me uneasy.

bwa index file not found

I checked, and S3://cgl-protect-data/hg19_references/bwa_index.tar.gz exists.
However, ProTECT throws this error:
[E::bwa_idx_load] fail to locate the index files
Full log at http://hgwdev.cse.ucsc.edu/~jeltje/toil/bwa.error
(note that the program is still running so the log doesn't end with an error)

Spawn mutect and radia using genome_fai

In case the user is not using the entire genome, spawn a job for each chromosome in the reference index.

"No Locks Available" error

reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,112 WARNING: toil.leader: K/p/job1Qif4A:  ---TOIL WORKER OUTPUT LOG---
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,112 WARNING: toil.leader: K/p/job1Qif4A:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,112 WARNING: toil.leader: K/p/job1Qif4A:  WARNING:toil.resource:Can't find resource for leader path '/neoantigen_prediction/protect/src/protect'
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,112 WARNING: toil.leader: K/p/job1Qif4A:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/neoantigen_prediction/protect/src', name='protect.pipeline.ProTECT')
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,112 WARNING: toil.leader: K/p/job1Qif4A:  CRITICAL:toil.job:CACHE: Unable to acquire lock on /mnt/isilon/maris_lab/target_nbl_ngs/Eric/neoantigen_prediction/test_outputs/toil-64121748-44a6-4d46-bbb4-b0216cfdcd85/cache-64121748-44a6-4d46-bbb4-b0216cfdcd85/.cacheLock
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,112 WARNING: toil.leader: K/p/job1Qif4A:  Traceback (most recent call last):
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,112 WARNING: toil.leader: K/p/job1Qif4A:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/worker.py", line 330, in main
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,112 WARNING: toil.leader: K/p/job1Qif4A:      fileStore = FileStore(jobStore, jobWrapper, localWorkerTempDir, blockFn)
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 995, in __init__
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:      self._setupCache()
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1469, in _setupCache
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:      with self._CacheState.open(self) as cacheInfo:
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:    File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:      return self.gen.next()
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1826, in open
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:      with outer.cacheLock():
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:    File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:      return self.gen.next()
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,113 WARNING: toil.leader: K/p/job1Qif4A:    File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1433, in cacheLock
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,114 WARNING: toil.leader: K/p/job1Qif4A:      flock(cacheLockFile, LOCK_EX)
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,114 WARNING: toil.leader: K/p/job1Qif4A:  IOError: [Errno 37] No locks available
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,114 WARNING: toil.leader: K/p/job1Qif4A:  Exiting the worker because of a failed jobWrapper on host reslnmarisig01.research.chop.edu
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,114 WARNING: toil.leader: K/p/job1Qif4A:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host reslnmarisig01.research.chop.edu
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,114 WARNING: toil.leader: K/p/job1Qif4A:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job K/p/job1Qif4A to 0
reslnmarisig01.research.chop.edu: 2016-07-01 11:31:46,114 WARNING: toil.leader: Job: K/p/job1Qif4A is completely failed

@hannes-ucsc any comments?

Modify spawn_mutect for CI

spawn_mutect calls run_mutect for each chromosome, but we only need chr6.

Update the Readme

Remove multiprocessing cpu count and use user-provided value instead

using dynamically calculated values will not work on a heterogeneous cluster. Hence we should make it a user-specified value.. for now. Till the fuzzy requirement PR is merged.

Implement Fusion support

Test Transvar