gwallison / openff Goto Github PK

This may become another branch of the integrated system

This is used in "location_cleanup.py" in builder_tasks

Unit testing is useful in limited situations for Open-FF, mostly for basic utilities. Probably more important are higher level functional tests that can be performed routinely

Functional tests: embedded in code and performed with every execution.

Unit type test: performed by pytest, usually before committing to repo.
If they are on this list, they have been started. Checked means completed.

• build.core.cas_tools
• common.nb_helper
• common.file_handlers

• remove from external_dataset_tools
• remove from Chemical_index
• remove from DataDictionary

The issue with notebooks, both in the browser and the stand-alone version is that they are not easily tested on Colab for functionality.

At the bare minimum, we should be able to run them straight through to make sure they work to completion.

Drafts in place and completed

• README
• Top
• What is FracFocus
• Open-FF overview
• Resolving chemical ID
• Calculating mass
• Standardizing text fields
• Proprietary records
• Generating the Open-FF data set
• The Open-FF data repository
• Getting openFF data
• external data in openFF
• Browser overview
• Generating the browser
• Notebook overview
• How to use notebooks

• in build_nb, add the necessary instructions and infrastructure to capture the ChemInformatics info for as many chemicals as possible. This should include Hazards (.xlsx and .sdf) and the safety data (.xlsx)
• At the minimum, put the presence/absence of the CI analysis into the bgCAS table so that downstream functions can access it.
• A more thorough treatment would be to create a third table, indexed by bgCAS but with all the info scraped from CI

While Open-FF has been able to find and flag these erroneous duplicates since 2019, a change in the data due to FracFocus V4 is undermining the ability to do that. The current workaround is to use archives to identify the extra records. That's not ideal for the long term. We are waiting for FracFocus to respond to the issue.

The CAS/Ing translation table has developed over years with much of it curated before I developed the current set of helper tools.

At some point it would be worthwhile to reanalyze the whole table with the new tools; especially those pairs that and not both matches.

FracFocus has occasionally announced that new features will be included, such as water source. This would be a change that we would want to detect as early as possible. Make a function to check if column names have changed at all after the initial bulk download is completed and flag if a change is detected.

• Move disclosure meta table maker to common
• Color values based on source (FF or OpenFF)
• Show important flags
• Explain why no mass was calculated, in those disclosures.
• Include explanation tables (not directly in the html)
• Can I make column-hiding work?
• Can I make floating header work?
• In DI of somewhere, make an API entry that will show and link to all disclosures
• For DI, because it is so big, point to (and make) county-based versions.
• Does it make sense to store the tables separately and 'include' the in a template?

In late Nov 2023, FracFocus started publishing their bulk data with a somewhat different format that includes renaming some fundamental fields (like 'UploadKey' -> 'DisclosureID'). Converting Open-FF to this new format will take some work to not break old code or make old repos unreadable.

• Refactor as you move
• Add unit testing where appropriate

Primarily molecule images (CompTox)

In the process of moving to this more integrated version of Open-FF, we want to refactor code to make it cleaner and more centralized. We will do this over the course of the process by taking sections at a time:

• file handling - builder tasks
• file handling - builder core
• file handling - notebooks
• file handling - browser