biocontainers / ci Goto Github PK

Use a scheduled job in jenkins for bioconda backup and security scan instead of a local server cron task

Better to get job status and view of what is going on

Record at regular interval anchore reports for images and store to biocontainers s3

In website, load from s3 instead of requesting to anchore

This will allow loss of anchore server data

(discussed with @mboudet today)
There are a few flaws that need to be adressed in the CI process (as implemented in https://github.com/BioContainers/ci/blob/master/github-ci/src/biocontainersci/biotools.py) that updates the metadata in the RSEc each time a new pull request is merged on the biocontainers containers repository:

Unique biocontainers filenames

We need to generate unique filenames for the biocontainers metadata files generated, e.g. instead of data/fastqc/biocontainers.yaml, https://github.com/research-software-ecosystem/content/blob/master/data/fastqc/fastqc.biocontainers.yaml. Here, the new filename pattern is data/[bio.tools ID]/[biocontainers ID].biocontainers.yaml. This will avoid collisions in case multiple containers refer to the same software in bio.tools, in which case any new container wrapping a bio.tools already packaged in another container would end replacing the contents of the previous file.

Generate files locally

biocontainers metadata files should be generated, at least as an option, in a local copy of the git repository, instead of creating a pull request, for easier testing.

Batch files generation

It would be practical to enable generating/updating metadata files for all the containers available in the repository, instead of only one, crawling all Dockerfile files in a local checkout of BioContainers/containers, and generating/updating the *.biocontainers.yaml files of a local checkout of research-software-ecosystem/content.

review metadata mapping

Have a exhaustive metadata review, to check that all metadata (at least LABEL, FROM, MAINTAINER) are mapped to the yaml file.

The current CI process only sends metadata about containers generated from Dockerfiles. We would need to do the same for all containers generated with mulled, so that the research-software-ecosystem/content repository has a file for each available mulled biocontainers as well.
Happy to take any suggestion on this.