Comments (15)
@alimanfoo Thanks so much for getting these ready! I'm hoping to get these test going early next week
from pipelines.
Hi @jessicaway, thanks so much for sharing the new shapeit4 runtimes. The results there look great, it looks like we should be able to use preemptibles, and we can reduce memory a lot too.
from pipelines.
Hi @jessicaway,
Here are the intervals for the gamb_colu phasing run:
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_2L_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_2R_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_3L_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_3R_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_X_200000_40000.txt
Here are the intervals for the arab phasing run:
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_arab_2L_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_arab_2R_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_arab_3L_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_arab_3R_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_arab_X_200000_40000.txt
from pipelines.
cc @gbggrant, @jessicaway, @kbergin
Re getting the input files (BAMs and zipped zarrs) into GCS, I'm thinking to copy them to our vo_agam_release bucket, which is the bucket we use for publicly releasing all data from Ag 3.0. That is a public bucket, so should be accessible to you for the phasing run, and would also be accessible to anyone else who wanted to work from the BAMs or zipped zarrs within Google Cloud. Does that sound OK?
from pipelines.
That sounds good to me. Thanks @alimanfoo!
from pipelines.
OK, I think all the inputs are in place ready for the phasing runs. Here's some info on where everything is.
Note that we want to perform three separate runs of the phasing pipeline, each with a different setup. I will call these three runs "gamb_colu_arab", "gamb_colu" and "arab". Here's an explanation of each of these runs:
- gamb_colu_arab - This phasing run uses samples from all three mosquito species represented in the Ag1000G phase 3 cohort (Anopheles gambiae, Anopheles coluzzii, Anopheles arabiensis).
- gamb_colu - This phasing run excludes any samples that are Anopheles arabiensis.
- arab - This phasing run includes only samples that are Anopheles arabiensis
Note that the arab run has the least number of samples (368), so you may want to start with this to get a sense of compute resources and cost, then move on to gamb_colu (2445 samples) then finally gamb_colu_arab (2814 samples).
Sample manifests
The sample manifests for the three phasing runs are on GCS. Each file is just a text file, one line per sample ID. Files are at the following URLs:
- gamb_colu_arab - gs://vo_agam_release/v3/snp_haplotypes/setup/phased_samples_gamb_colu_arab.txt
- gamb_colu - gs://vo_agam_release/v3/snp_haplotypes/setup/phased_samples_gamb_colu.txt
- arab - gs://vo_agam_release/v3/snp_haplotypes/setup/phased_samples_arab.txt
Sites
The sets of sites and alleles to be phased are at the following URLs:
- gamb_colu_arab - gs://vo_agam_release/v3/snp_haplotypes/setup/phased_sites_gamb_colu_arab.zarr.zip
- gamb_colu - gs://vo_agam_release/v3/snp_haplotypes/setup/phased_sites_gamb_colu.zarr.zip
- arab - gs://vo_agam_release/v3/snp_haplotypes/setup/phased_sites_arab.zarr.zip
Alignments (BAM files)
BAM files for all samples are in the vo_agam_release bucket. A CSV file containing a mapping from sample identifiers to GCS URLs is available here:
gs://vo_agam_release/v3/alignments/catalog.csv
Genotypes (zipped zarr files)
Zipped zarr files with SNP genotypes for each sample are in the vo_agam_release bucket. A CSV file containing a mapping from sample identifiers to GCS URLs is available here:
gs://vo_agam_release/v3/snp_genotypes/per_sample/catalog.csv
Recombination maps
I've created new recombination map files, one for each chromosome arm. These should be in the shapeit gmap format, but let me know if any problems. Files are here:
- gs://vo_agam_release/v3/snp_haplotypes/setup/2R.gmap
- gs://vo_agam_release/v3/snp_haplotypes/setup/2L.gmap
- gs://vo_agam_release/v3/snp_haplotypes/setup/3R.gmap
- gs://vo_agam_release/v3/snp_haplotypes/setup/3L.gmap
- gs://vo_agam_release/v3/snp_haplotypes/setup/X.gmap
The same recombination maps can be used for all three phasing runs.
I think that's all the inputs but let me know if I've missed anything :-) Cc @gbggrant @jessicaway @kbergin @seretol.
from pipelines.
Hi @jessicaway, re sample metadata for terra, here is a notebook illustrating how to load sample metadata for all samples in Ag1000G phase 3. Also I'll attach the CSV export here: ag3_samples.csv.
from pipelines.
Thanks @alimanfoo! I will get this added to the workspace
from pipelines.
Surfacing discussion about next steps from this week's call.
We've decided to make changes for the gamb_colu_arab run to see if we can reduce costs and get predictable runtimes and resource requirements, by (1) using intervals with a fixed number of SNPs, and (2) upgrading SHAPEIT4 to version 4.2.1.
Plan:
- Run the gamb_colu_arab set through read-backed phasing
- Analyse runtime data from the gamb_colu run and decide intervals to use for gamb_colu_arab run
- Upgrade phasing pipeline to use SHAPEIT4 v4.2.1
- Do a test run of SHAPEIT4 v4.2.1 on one of the new intervals to determine expected resource requirements
- Set up the full gamb_colu_arab run, using new pipeline (with upgraded SHAPEIT4), new resource configuration, and new intervals lists
- Kick off the full gamb_colu_arab run
@jessicaway, @kbergin, @gbggrant does that look right to you?
from pipelines.
Here's the gamb_colu runtime plotted against the number of sites:
Based on this I'm tempted to go for 200k sites intervals, there should still be a decent amount of information there in each interval, and runtimes should be well within 24hrs, so we could use preemptibles.
from pipelines.
Hi @jessicaway, I've created intervals files for 200k SNPs intervals with 40k SNPs overlap. Here are the intervals for gamb_colu_arab:
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_arab_2L_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_arab_2R_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_arab_3L_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_arab_3R_200000_40000.txt
- gs://vo_agam_release/v3/snp_haplotypes/setup/intervals_gamb_colu_arab_X_200000_40000.txt
When choosing a few intervals to use for testing and investigating resource requirements, I'd suggest to pick an interval from somewhere near the middle of each chromosome, rather than the ends. I am hoping that the memory requirements will be nearly constant across all intervals, but the runtime may vary a bit if there is a lot more recombination (and hence less haplotype sharing) in some intervals versus others. By picking intervals near the middle of chromosomes, that should avoid the regions of low recombination near the centromere or telomere, and so give us a better sense of what the maximum resource requirements should be. I'd suggest to do a test run on maybe three intervals, from different chromosomes?
from pipelines.
@alimanfoo, of course! We are also very excited by the new results! We will add the preemptibles and back off the memory (and probably reduce the disk as well). I expect this run will be much cheaper
from pipelines.
Hi @jessicaway, sorry it took a bit longer than expected to check the gamb_colu_arab
outputs. There was an unexpected thing where the output haplotypes for the crosses samples were not fully concordant with the input genotypes, which had me stumped for a while. Eventually I realised that these samples were not included when I ascertained the biallelic sites. I think what is happening is that in these samples, there are some additional alleles at the phasing sites, and so genotypes get converted to missing at the start of the pipeline, but then shapeit4 imputes missing genotypes, and so fills them in with something different. In any case I think this is expected behaviour, and everything else about the outputs looks good, so please go ahead with the gamb_colu
and arab
phasing runs.
from pipelines.
Thanks @alimanfoo! I will run the gamb_colu
and arab
sample sets as discussed
from pipelines.
Closing as completed 🌸
from pipelines.
Related Issues (20)
- Provide example crams released by Core Sequencing HOT 3
- Define read group contents in vector short read alignment pipelines HOT 1
- Input source for mosquito short read alignment pipeline HOT 3
- Validation/truth datasets for variant calling calibration, comparison and evaluation HOT 26
- Copy script to convert vcf to zarr to this repo
- Port baseline Plasmodium amplicon SNP calling pipeline. HOT 1
- Add platform argument to build_pipeline_release script HOT 2
- How to make bwa short read alignment deterministic for testing HOT 1
- Mosquito phasing pipeline development HOT 22
- Genotype Concordance Validation of genotyping pipelines HOT 10
- Create pipelines that download multiple samples from IRODS in a batch
- Determine final lanelet manifest fields required for vector alignment and genotyping HOT 1
- Cromwell configurations on farm5 are using localization = copy, which explodes size of workflow working directory
- Convert cram to bam in vector alignment pipeline
- malaria-dk lsf group deprecated
- Output of vector short read alignment pipeline is named *.bam.bam HOT 1
- Use flexible fields in manifest
- Validate results for phasing pipeline HOT 17
- Number of SNPs and Indels doesn't match with the numbers reported in README files HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pipelines.