con / tinuum

Resources (later might be a tool) to create reproducible computational environments

tinuum

Resources (later might be a tool) to create reproducible computational environments.

At the moment it is just a list of issues to collect information on ways to assist with specifications to create reproducible environments, e.g. within containers, so that rerunning provisioning script creates identically in functionality, if not bit-wise, environment images.

tinuum's Issues

PyPI (pip)

https://pypi.org/project/pypi-timemachine/

Debian / NeuroDebian

• https://snapshot.debian.org/
• http://snapshot-neuro.debian.net/ (TODO: configure https) - fork of above

• https://github.com/neurodebian/neurodebian/blob/master/tools/nd_freeze to simplify configuring

Nix

There is a way as was mentioned by @matrss in @distribits 2024 chat discussions, where on the example of https://atris.fz-juelich.de/m.risse/datalad-nix-demo/src/branch/main/flake.nix#L5 where for nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable"; @matrss elaborates:

Yes, you can also specify a specific commit hash (but I think it would be called rev=<hash> then, instead of ref=, it's documented here: https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake.html#flake-references). Of course it also doesn't have to be github, it can be gitlab or any git repository available over http, or even another local directory and stuff like that.
I should add that nix flakes are still considered an experimental feature (but it has been working well for a long time now). All of this can be done with non-experimental nix as well, but it is a bit less standardized what it would look like.

Gentoo

Following the setup done by @TheChymera for https://github.com/con/opfvta-reexecution/blob/master/code/images/Containerfile#L11 with a little cleaned up setup in https://github.com/TheChymera/neurodocker_g/blob/master/Containerfile and which hopefully would eventually be generalized within

• ReproNim/neurodocker#611

Review Article

@yarikoptic you mentioned in the meeting today that you envision this becoming a resource of available reproducible computational technologies, and possibly grow into a review article.

A few questions:
1. Is an all-encompassing overview really relevant for our work? We're mainly using docker/singularity
2. How would this be different from the write-up in opfvta-reexecution?
3. I think in order to define a scope we might want to start with a draft title, do you have any suggestions?

GNU Guix

Guix is a package manager, package repository, and through guix system a GNU/Linux distribution that is a different implementation of the same functional packaging approach that nix (#5) first introduced.

To get an isolated environment with some packages available you can use guix shell: https://guix.gnu.org/manual/en/html_node/Invoking-guix-shell.html

To get a reproducible isolated environment you just have to combine guix shell with guix time-machine (and probably checking in a channel description file into you project): https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html

These two blog posts go into more detail on how this could work:

• https://www.futurile.net/2023/04/30/guix-reproducible-dev-environments/
• https://www.futurile.net/2023/10/17/guix-time-travel-dev-environments/

There is even an active effort to make Guix on HPC a great experience: https://hpc.guix.info/

(Disclaimer: I haven't used guix myself yet; for now I have stayed with nix and just watched guix from afar. But I consider the functional packaging approach implemented by both to be the future of package management, no matter which implementation will become more popular. It is just so intuitive to do it this way.)

conda/conda-forge

Responses/hints got so far on gitter (thanks!)

@bollwyvl

there was a hosted one at some point, but it was not feasible to maintain

one could theoreticallty do such a thing by manipulating a local cache of repodata, but has gotten harder with recent changes (jlap, etc)

@jaimergp

Check https://github.com/conda/conda-libmamba-solver/blob/main/tests/repodata_time_machine.py. With CEP-15 this will be better supported once implemented. For now I think it's only valid to generate --dry-run --json payloads which you could turn into a @EXPLICIT style lockfile.