Comments (3)
A related question is how we deal with the platform-agnostic versus platform-specific parts of a pipeline.
E.g., @kbergin, @gbggrant, mentioned that the approach taken at Broad is to separate pipelines so you have a core WDL providing a platform-agnostic part, and then a wrapper WDL which provides the necessary config and setup for running in a particular compute environment, and which calls the inner WDL.
I think the idea for this repo is that we would put the core platform-agnostic parts of pipelines here, with the goal that other teams could re-use them to run malariagen pipelines on their own compute infrastructure. But then where do we put the platform-specific wrappers? Here or a different repo?
from pipelines.
Straw man proposal:
We keep this repo for only platform-agnostic pipeline code. We should be able to submit pipelines written here to dockstore.
We create a separate github repo for the Sanger platform-specific pipeline wrapper code. Maybe something like malariagen/pipeline-wrappers or something like that. That could be a private repo.
To be figured out, how to conveniently connect the two repos during development. I.e., whoever is doing pipeline development may need to work with both repos at the same time, because there may need to be changes at both levels.
from pipelines.
I like this approach, having platform-agnostic pipeline code would allow us to test the pipeline in other environments, which could speed development and get all of the unexpected platform-dependencies out.
from pipelines.
Related Issues (20)
- Provide example crams released by Core Sequencing HOT 3
- Define read group contents in vector short read alignment pipelines HOT 1
- Input source for mosquito short read alignment pipeline HOT 3
- Validation/truth datasets for variant calling calibration, comparison and evaluation HOT 26
- Copy script to convert vcf to zarr to this repo
- Port baseline Plasmodium amplicon SNP calling pipeline. HOT 1
- Add platform argument to build_pipeline_release script HOT 2
- How to make bwa short read alignment deterministic for testing HOT 1
- Mosquito phasing pipeline development HOT 22
- Genotype Concordance Validation of genotyping pipelines HOT 10
- Create pipelines that download multiple samples from IRODS in a batch
- Determine final lanelet manifest fields required for vector alignment and genotyping HOT 1
- Cromwell configurations on farm5 are using localization = copy, which explodes size of workflow working directory
- Convert cram to bam in vector alignment pipeline
- malaria-dk lsf group deprecated
- Output of vector short read alignment pipeline is named *.bam.bam HOT 1
- Use flexible fields in manifest
- Validate results for phasing pipeline HOT 17
- Ag3.0 phasing pipeline production run HOT 15
- Number of SNPs and Indels doesn't match with the numbers reported in README files HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pipelines.