port CombineVariants about gatk

[xiucz]

Please port CombineVariants. It is really really a good tool.

[droazen]

I wonder if users mainly just want something that adds the "set" annotation, so that you can do things like -select 'set == "Intersection"' in SelectVariants...

[ldgauthier]

I didn't want to port CombineVariants because it has some really insidious behavior when it comes to choosing attributes that differ between/among the variants being combined. I did have to use it myself recently, so I appreciate its utility. Someone suggested I use bcftools merge instead. When I did I found that bcftools was at least two orders of magnitude faster and addressed my use case perfectly. Is there CombineVariants functionality that is not represented in bcftools?

[xiucz]

@ldgauthier
Hi, can you share the bcftools merge code for different callers' results here? Thanks a lot.

[akiezun]

the requirement is to port the CombineVariants tool and tests. Current tests use Broad-visible data and that data needs to be reviewed and shared publicly if possible.

[davidbenjamin]

@lbergelson Can we close this issue since we now use MergeVcfs in the picard package? 3 1/2 years is a long time to be high priority. . .

[bhanugandham]

Users have shown interest in porting CombineVariants to GATK4. Features such as, combining vcfs with variants present in all or a fraction of samples exists in CombineVariants but not in MergeVcfs/SelectVariants. It would be helpful to look into this.

[ldgauthier]

The behavior of the GATK3 CombineVariants was very inconsistent and the arguments weren't entirely clear. I also suspect that some operations weren't possible with the arguments given. Rather than port that old broken version, I would advocate for an overhaul or rewrite.

@bhanugandham it's going to be a big project to collect requirements and expected behavior for this tool. For example, what should the MQ be for the combined VCF for two different input VCFs with different MQ values? Much of the confusion stemmed from the old ability to merge VCFs containing the same sample. In the case where we take one genotype for each sample name (e.g. the old -genotypeMergeOptions PRIORITIZE) then I believe the old behavior was wrong in some cases, taking the filter status from an input VCF at random. We also need to clarify FilteredRecordMergeType options, e.g. https://github.com/broadinstitute/gsa-unstable/issues/935

[bhanugandham]

@ldgauthier what do you suggest the course of action should be? I mean for when users ask for certain feature that were in CombineVariants but not in MergeVcfs? Should I just create a new issue for each argument request?

[bhanugandham]

I have asked the users who requested porting CombineVariants for specific features that do not exist in MergeVcfs or other tools in gatk4, and are useful to them.
I will post results from that here, that should help make a decision.

[pieterlukasse]

In #2489 @vdauwera mentions that CombineVariants will not be ported, but in this ticket @bhanugandham 's comment makes it sound like this decision is not final yet. We would like to use the functionality of CombineVariants in a new pipeline we are implementing. What is our best option? Should we use the old GATK 3?

[bhanugandham]

@pieterlukasse we are not looking into porting CombineVariants. If you have to use CombineVariants, it is available in the old GATK3 version. However, unfortunately we will not be able to provide support it.

[1onic]

Please do!