nf-core / variantbenchmarking

A nextflow variant benchmarking pipeline - premature

Introduction

nf-core/variantbenchmarking is a bioinformatics pipeline that ...

1. Standardization of SVs in test VCF files
2. Normalization of SVs in test VCF files
3. Normalization of SVs in truth VCF files
4. SV stats and histograms
5. Germline benchmarking of SVs
6. Somatic benchmarking of SVs
7. Final report and comparisons

Usage

Note

If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.

Supported SV callers: Manta, SVaba, Dragen, Delly, Lumpy .. Available Truth samples: HG002, SEQC2

• If you have unresolved SVs, it is recommended to use only truvari with --pctsim 0.

• The size filtration parameters provided by truvari and svbenchmark do not apply in the same way. That is why using them is not recommended through the pipeline, but variants can be filtered safely in vcf normalization steps.

• Please note that it is not possible to use exactly the same parameters for different benchmarking methods.

Now, you can run the pipeline using:

nextflow run nf-core/variantbenchmarking \
   -profile <docker/singularity/.../institute> \
   --input samplesheet.csv \
   --outdir <OUTDIR>

Warning

Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.

For more details and further functionality, please refer to the usage documentation and the parameter documentation.

Pipeline output

To see the results of an example test run with a full size dataset refer to the results tab on the nf-core website pipeline page. For more details about the output files and reports, please refer to the output documentation.

Credits

nf-core/variantbenchmarking was originally written by [email protected].

We thank the following people for their extensive assistance in the development of this pipeline:

Contributions and Support

If you would like to contribute to this pipeline, please see the contributing guidelines.

For further information or help, don't hesitate to get in touch on the Slack #variantbenchmarking channel (you can join with this invite).

Citations

An extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.

You can cite the nf-core publication as follows:

The nf-core framework for community-curated bioinformatics pipelines.

Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.

variantbenchmarking's Issues

vcfstats module

Description of feature

investigate vcfstats to see if it is a good way to visualize vcf statistics

https://pypi.org/project/vcfstats/

add SQLite database

Description of feature

write an app to write benchmarking results into a SQLite database file

URGENT: pin nf-validation version

Description of the bug

To prevent breaking this pipeline in the near future, the nf-validation version should be pinned to version 1.1.3 like:

plugins {
    id '[email protected]'
}

Command used and terminal output

No response

Relevant files

No response

System information

No response

add vcf_preprocessing_deduplication subworkflow

Description of feature

add bcftools norm -deduplciation part as subworkflow

add a script to merge benchmarking results

Description of feature

This workflow reports benchmark results of different tools from a same sample. it would be nice to have a summary table comparing different tools.

Benchmarking tools differ in their reports, a script to merge different results can be useful. The tables then can be used for plotting as well.

add config for svaba

Description of feature

SVABA results does not include SVLEN, generate config file in svync to calculate SVLEN.

add SURVIVOR merge

Description of feature

SURVIVOR merge can be used to compare multiple vcf files

• Comparing evaluation results?
• TP results for input tools?

svync process is not running through

Description of the bug

I think there is an error in channel construction step that svync process is not running. Can you please check it @nvnieuwk ?

Command used and terminal output

executor >  local (1)
[cb/395e11] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:SV_VCF_CONVERSIONS:BGZIP_TABIX (manta)         [100%] 2 of 2, cached: 2 ✔
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:SV_VCF_CONVERSIONS:SVYNC                       -
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TEST:BGZIP_TABIX                  -
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TEST:BCFTOOLS_REHEADER_TEST       -
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TEST:TABIX_BGZIPTABIX_1           -
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TEST:VCF_VARIANT_FILTERING:TAB... -
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TEST:VCF_VARIANT_FILTERING:SUR... -
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TEST:VCF_VARIANT_FILTERING:TAB... -
[c6/ff4e82] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TRUTH:BGZIP_TABIX (HG002)         [100%] 1 of 1, cached: 1 ✔
[cd/a28218] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TRUTH:BCFTOOLS_REHEADER_TRUTH ... [100%] 1 of 1, cached: 1 ✔
[da/a2111e] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TRUTH:TABIX_BGZIPTABIX (HG002)    [100%] 1 of 1, cached: 1 ✔
[3d/bf7ee7] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:REPORT_VCF_STATISTICS:SURVIVOR_STATS (HG002)   [100%] 1 of 1, cached: 1 ✔
[26/0962bc] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:REPORT_VCF_STATISTICS:BCFTOOLS_STATS (HG002)   [100%] 1 of 1, cached: 1 ✔
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:SV_GERMLINE_BENCHMARK:TRUVARI_BENCH            -
[-        ] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:SV_GERMLINE_BENCHMARK:SVANALYZER_SVBENCHMARK   -
[79/810842] process > NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:MULTIQC                                        [100%] 1 of 1, failed: 1 ✘
Execution cancelled -- Finishing pending tasks before exit

Relevant files

No response

System information

No response

standardization VCF filtering

Description of feature

1. small variant benchmarking (easier):

• pass_only
• MAPQ

• QUAL
  

2. sv benchmarking (harder):

• FILTER: different IDs
• IMPRECISE
• MAPQ
• EVENT

dragen report nonstandard variants

Description of feature

DRAGEN reports NN included variants:

chr21 44888040 DRAGEN:INS:16918:1:1:0:1:1 CNNNNNNNNNN CCCTCACTCACCACATACACACCAACCACCACATACCACACACACCACACCACGCACCACCCCATGTCACACACACATCACATCACACACACCACACGCCACACCACACACACCTCACACCCAAATCACATATCACACACAGACACCACATACCTCACAACACACATCATAACACCACACACACCATATATCACACACACACACACACCACATATTACACACCACACATATCACACTACACACCTCATACCACACACCATATACCACACACACGGCACCAAATATCACCCACACATCACACAACACATACTTCACACATACACCACATACCCCACACCACACA 999 PASS END=44888050;SVTYPE=INS;SVLEN=342;CIGAR=1M342I10D GT:FT:GQ:PL:PR:SR 1/1:PASS:64:999,67,0:6,15:0,42

causing svbenchmark error.

wittyer module

Description of feature

Push wittyer to nf-core/modules

example: https://github.com/nf-core/modules/tree/master/modules/nf-core/bcl2fastq

update to [email protected]

Description of feature

nf-core/sarek#1478
nf-core/rnaseq#1287
nf-cmgg/germline#174

add plot functions

Description of feature

Plotting functions will be added for both small and indel variant benchmarking.

Find the example script here: https://github.com/christopher-hakkaart/bench/blob/dev/bin/sv_report.r

exposing benchmarking params

Description of feature

Benchmarking for SVs highly depends on parameters like:

truvari: pctsim, pctseq, minsex,maxseq, --dup-to-ins, extend...

svbenchmark:
https://svanalyzer.readthedocs.io/en/latest/svbenchmark.html

It is not easy (sometimes impossible) to align those params but it is also not easy to find a default value for them! That is why I think it might be better to expose all those in config.

add test case hg38 for sv

Description of feature

HG002 is run for nf-cmgg-structural using hg38. Chr21 is subtracted and placed into https://github.com/kubranarci/benchmark_datasets/tree/main/nf-cgmm-structural/chr21.

implement SEQC2 data

Description of feature

test germline benchmark with SEQC2 data available here:https://genomebiology.biomedcentral.com/articles/10.1186/s13059-022-02816-6

survivor stats does not work with truth

Description of the bug

survivor stats does not work with truth becouse of unproper id issue

Command used and terminal output

No response

Relevant files

No response

System information

No response

split variant benchmarking by type

Description of feature

make it possible to benchmark only by certain type

• small variants --> snv,indel,sv
• svs variants --> ins,dup,inv ...