cancervariants / gene-normalization Goto Github PK

Creating concept groups is slow and creating concept groups in production environment is even slower. We should look into speeding this up.

• NCBI
• Ensembl

The normalize endpoint should generate a single, merged concept for search terms.

For elastic beanstalk

• Ensembl: biotype

We currently add a GSI on concept_id in #97 . However, we should see if we're able to use concept_id as the partition key and label_and_type as the sort key to prevent the extra creation of a GSI. Did not do this in #97 due to interest in time

We currently only use the non alternative loci set. We should also include the alternative loci set from the download page.

Not just the strongest match per source

NCBI uses 3 different files (history, info, gff). History and info data are updated daily, but gff data is versioned by assembly. We currently use the timestamp at which we retrieve the data (we should also fix this so that it's the timestamp from the ftp site). I think we should consider storing metadata for each file. Also, the current source meta does not indicate the files used and instead points to the ftp site.

The merged concept for hgnc:37133 has alternate_labels: "A1BGAS", "FLJ23569", "NCRNA00181", "A1BG-AS". Querying these alternate_labels returns different match_type scores, when they theoretically should return the same score.

• Better documentation
  • Add type hints
• DRY
• Remove unused code
• Rather than using vrs-python's VRS models, use ga4gh.vrsatile.pydantic models in gene.vrs_locations
• Check if we can remove EBSampleApp-Python.iml?
• Add flake8-annotations + double quotes
• String enums in schemas

Some models/fields have been renamed or deprecated

Our EB currently uses python 3.7. We should upgrade to 3.8.

https://www.genenames.org/download/statistics-and-files/

NCBI has retired gene identifiers in the past, e.g.:

• ncbigene:401317 now maps to ncbigene:9586. Our normalizer should match the old ID to the current record. This should be treated analogous to the "previous symbols" attribute in HGNC.
• ncbigene:103344718 is a discontinued gene. We should normalize to concepts like this, but also have a status attribute that makes it clear this is no longer considered a gene. We should also emit a warning in our warnings attribute for each such entry, akin to: ncbigene:103344718 is a discontinued gene concept.

When specifying locations, we should use VRS Location objects.

ChromosomeLocation for the ISCN-style entries in the HGNC "location" field

SequenceLocation for the Chr/Start/Stop entries from ensembl.

This should reduce the following attributes:
seqid
start
stop
strand
location

down to:
location: (VRS Location)
strand: enum( '+', '-', Null)

@jarbesfeld 's GH Actions in py-gene-fusions are failing due to our schema classes

Forgot to update schema examples to reflect VRS/VRSATILE updates

@jarbesfeld will be using these models in py-gene-fusions

Add an option to CLI to use local files rather than downloading from the source's FTP site

A docker container would be useful

Switch to downloading files from FTP sites

https://www.ncbi.nlm.nih.gov/home/download/

This will help with going serverless

Consider creating sample data to test ETL methods. If we don't go this route, we should clean up the current test data

• Set use_enum_values in pydantic model config
• Change response_datetime to str

Separate between those representing gene concepts from those representing associated concepts.

https://uswest.ensembl.org/info/data/index.html

We currently have the production database as the default, but we should switch to using the local database.

Some helpful posts:

• https://aws.amazon.com/blogs/compute/creating-a-scalable-serverless-import-process-for-amazon-dynamodb/
• https://medium.com/swlh/processing-large-s3-files-with-aws-lambda-2c5840ae5c91
• https://aws.plainenglish.io/using-aws-lambda-with-python-to-auto-write-to-a-dynamodb-table-from-s3-ee01da85094

If not all sources are loaded in the database, the queries on the search endpoint will fail.

https://pypi.org/project/ga4gh.vrsatile.pydantic/

We had been using vrs-python models for validation. The addition of validators being used in schemas are now causing pydantic validation errors when loading sources