neherlab / nextclade_data_workflows

Checking new tree

1. Download generated files into nextclade data workflow repo:

   scp -rC [email protected]:~/nextclade_data_workflows/sars-cov-2/output output

2. Plug them into nextclade.org advanced view.

3. Filter to new nodes and check that:

   • clades are clean
   • no big outliers

4. Check tag.json is up to date (ideally update in profiles/tag.json for posterity)

5. Check qc.json does not regress (ideally update in profiles/qc.json for posterity) [beware, codons are 0 indexed]

6. Potentially run scripts/common_stops.py and scripts/common_frameshifts.py to add new stops/frameshifts that have become more common to qc.json

Identifying most common frame shifts and stop conds

1. Download metadata to data/metadata_raw.tsv

2. Run snakemake workflow with following commands/targets:

   snakemake --profile=profiles/clades pre-processed/frameshifts.tsv -R select_frameshifts
   snakemake --profile=profiles/clades pre-processed/stops.tsv -R select_stops

3. Format most commons stops/fs into qc.json JSON format using

   python3 scripts/common_stops.py
   python3 scripts/common_frameshifts.py

4. Manually check resul for plausibility and add to qc.json

Committing to data repo

1. Go to nextclade_data_workflow repo

2. Checkout branch, open PR to master

3. Copy output from workflow repo to data repo

   cp -r output/sars-cov-2/references/MN908947/versions/  ../../nextclade_data/data/datasets/sars-cov-2/references/MN908947/versions

4. Update changelog.md

5. Get Ivan to review

6. Merge into master

Release process

Follow release guidelines as outlined here: https://github.com/nextstrain/nextclade_data#dataset-release-process

nextclade_data_workflows's Issues

Rename `gene` to `cds`

We need to make workflows output the right names as required by: nextstrain/nextclade#1376

Might be possible to do it almost automatically via s/find/replace

colour omicron in tree properly

MPXV: Remove duplicates (after masking) to reduce tree size

Removable are 0 divergence below internal node - save one sequence that should be kept.

ENH: Allow recombinant sublineages to be present

For a start, can treat recombinant sublineages as recombinants of their own - but it would be better if we built separate trees for them, root being on parent

ENH: Put all recombinants on single branch from root

To combat ancestral reconstruction inferring S:614G at root, I can add all recombinants on a single subtree from root - as opposed to them being added all individually on root.

Right now having long branch length works, but in the future with more recombinants this may not be enough.

Unmask S:95

Don't pull GISAID sequences into output

Filter out Egypt/Iraq due to often bad quality

For some reason despite exclude-where countr=Egypt is still in the subsample.

Need to debug.

Don't count nucleotides 0-100 towards divergence

44 is very flip-floppy due to being often not properly sequenced.

We should not show it in calculation of divergence to avoid flip flops from lengthening the tree

Nextclade distance calculation should exempt 0-100 from distance calculations as well, but that's a different feature (requires an addition to virus_properties.json though - masked sites for placement)

Auto generate constraint.nwk from pango alias file, allow additional resolution

Constraint nwk is currently hand written.

Would be much better to automate this and allow extra granularity when known.

Maybe via yaml config file as opposed to nwk

Include XBB in 21L build to not call BJ.1

Use hand curated list of high quality Omicron sequences

ENH: Soft mask 26577 (M:19) instead of hard mask

ENH: Remove real sequences from Nextclade build

Prohibit reversions by default, make allowlist for real reversions

As byproduct of synthetic Pango, output all detected reversions

As a second step: only allow reversions in a whitelist - requires review of potentially new true reversions

Use pango defining mutations to improve synthetic lineage curation

Use the following list as mutations to always include:
https://www.pango.network/summary-of-designated-omicron-lineages/

Include GISAID data provenance

Exclude S:142G sequences within Delta as outliers

We know all of Delta should have 142D -> remove offending sequences

ENH: Make BA.2 build side-product of main build

Right now we have the builds on separate branches. It would be nice to produce them all in one workflow to make maintenance easier.

The BA.2 build forks after tree building: it's a clean subtree from the common ancestor of BA.2 and BA.3, rooted on BA.2

ENH: Add recombinants back in, as subtree connected to root

For each recombinant, make a separate subtree (including the root), and connect it at the root.

This way we don't generate non-biological internal nodes that could attract attachments that we don't want to have (e.g. somewhat reverted B.1.1.7 attaching at internal parent node of B.1.1.7 and XB).

ENH: Build proper tree for XBB.1.5 rather than the simplistic current one

Right now, all recombinant trees are built very simplistically, based on Pango hierarchy. This presents issues now that XBB keeps growing with more lineages.

Way to go is to build the XBB tree separately, rooted on the reconstructed founder, then do ancestral reconstruction with the founder attached to just Wuhan/BA.2 (depending on tree), then graft the result over to the other tree after reconstruction (to prevent recombinants from messing up the reconstruction).

BUG: 28370 not overwritten with `-` but `T` 28371 -> T in BF.7

Overwrite only works for reversions to reference?

Check raw BF.7 and algorithm

ENH: Add date of designation to SC2 build for pango lineages

Request: Add vaccine strains and clade defining viruses to NextClade flu dataset

Hi there,

On NextClade tree view, it would be very useful to include and mark (if possible) vaccine strains and clade defining viruses.

For vaccine strains, similar to Nextstrain portal, it would be great to include an X in the tree view of NextClade. At the minimum including the virus strains would be very useful.

Clade defining viruses are a little bit more tricky. Not only do clades continually get added, but potentially viruses which define these clades could change. This is related to nextstrain/seasonal-flu#66. I can provide a list of clade defining viruses to add to NextClade, at minimum this would be super useful as the user can manually search for the clade defining viruses.

Thanks,

Ammar

ENH: Manual overwrite of lineage mutations

Fix dropout problems by hand curation.

Could autodetect issues in final tree: BA.5 not basal to where BA.5 designation tarts

ENH: Add escape score etc as colorings

Add colorings:

• Escape score
• ACE2 score
• Composite score
• Growth advantage modelled by Wenseleer, Gerstung, Murrell models
• Date of 5th (non misdated) sequence or something like this
• Total reversions
• Diff of reversions
• Number of S mutations
• Diff of S mutations
• Key RBD mutations
• Diff of key RBD mutations
  ...

If you have further ideas, feel free to comment

MPXV: Add reversions to virus_properties in B.1 build

ENH: Attach recombinants at most recent common ancestor

This way we don't have the artificial non-specific clade "recombinant"

For example, XBB is derived from 21L and 22D - 22D derives from 21L, so XBB should be 21L, and attach at 21L, or a more recent common ancestor.

Importantly, the branch length to recombinants needs to be long, so that the ancestral reconstruction is not misled.

It would be nice if it was possible to visually indicate recombinants in Auspice, e.g. by having dashed line instead of full line.