andersen-lab / bjorn_utils Goto Github PK

Need to generate latest metadata and send to SRA/Biosample process to upload

Changes in non-coding regions are currently moving most of Delta sequences to inspect folder. Mark suggests altering this such that

• Changes in NC regions do not cause inspect flags to be thrown
• Parameters for allowing the above are included in pipeline run

Will further consult with Mark and Karthik G. before taking out a branch against this issue.

Ideally this should be a passable property so the pipeline can be run in other places

Simply environment file to have only those dependencies required for bjorn_utils

Current date format gets changed by excel upon viewing, correct this so that excel knows what format the date should be in.

Currently the release workflow requires manually interacting with the GISAID metadata tab. This issue and accompanying PR suggest these changes.

• Get latest version of metadata from GISAID tab
• Get latest version of metadata from GitHub
• Based on coverage and variation cutoffs, push changes to GISAID tab specifically columns C & D

Will review these further before taking out new branch with Mark and Karthik G.

Add following potential filters that work in an automated way to flag sequences during release pipeline:

Metadata filters

• Missing Metadata
• Unlikely Collection Data

Sequence filters

• Unlikely date for lineage
• Violation of molecular clock

This will likely need to be sub-issued during implementation

Automated and combine the listed steps to reduce command overhead when generating output files

Example commands:

# inspect suspicious_mutations.csv
Combine msa/*_white.fa and *_inspect.fa
# Convert aligned to unaligned fasta
bash ~/code/bjorn/scripts/convert_to_unaigned_fasta.sh 2021-08-25_release_aligned_combined.fa > 2021-08-25_release_unaligned_combined.fa
# Convert multifastas to sindividual fasta preferably in a subfolder
bash ~/code/bjorn/scripts/multifasta_to_fasta.sh ../2021-08-25_release_unaligned_combined.fa

Need just 1 bam folder and 1 MSA folder - nothing else for the bjorn output

This currently sits in a notebook - need to update so we can generate the files that need to be uploaded and send it over to the ncbi_batch_push repo for action

Adjust output so that samples column is easier to read

Additionally, need a try: except clause since sometimes these data frames can be empty

Get data from these three sources and add to the metadata file

Need to automate the upload of files to these different sources

GISAID

• Need to fix header errors
  Github
  Google Cloud
• No need to wait for analysis to be complete to start this. In a fully automated pipeline, Server can start uploading these files right away
  NCBI
• To to solve backlog of uploads and batch our current set
  SD County
• Need to autogenerate log file to send to SD county so that they can get sequence data to the state. Should be straightforward from log that is generated.

Will review each of these portions with Mark and Karthik G. before taking out branch against them.