o2r-project / containerit Goto Github PK

As an alternative to Docker, see https://www.vagrantup.com/docs/vagrantfile/

• http://www.vagrantbox.es/
• https://atlas.hashicorp.com/ljw/boxes/arch-scicomp
• https://atlas.hashicorp.com/jmbejara/boxes/jb-stack
• https://atlas.hashicorp.com/quickstart/boxes/cdh_r

Expands on an idea mentioned in #6 (comment)

After a session (or workspace or something else) was containerized, a user may be able to test-build an image in order to verify that

(1) The build is successful
(2) The local session matches the dockerized session
(3) More ideas?

1 and 2 can be achieved analog to tests/testthat/test_sessioninfo_reproduce.R, by turning the test into feature

https://github.com/codemeta/codemeta/pull/97/files

Should be quite straightforward thanks to https://github.com/ropensci/jsonld

See codemeta/codemeta#122

• create codemeta for session
• create codemeta for rmarkdown document
• put codemeta json in a label

Extends #37

Simple R images like rocker/r-ver do not come with a GUI, therefore, the use of R is restricted to the console.

With such an configuration (try for instance docker run -it --rm rocker/r-ver) users are not able to view R plots or any file or data that cannot be printed directly to the console.

Therefore, it would be beneficial to leverage Rstudio images (see rocker/rstudio) and thus restore sessions directly in an Rstudio Server session.

See https://github.com/rocker-org/rocker/wiki/Using-the-RStudio-image

Write a function that pulls a Dockerized linter (projectatomic/dockerfile-lint looks good) container and executes it on a given path and shows the output on the R console.

As becomes clear in the discussion on geospatial libraries in Rocker, the versions of linked external libraries matter.

Can we support packaging explicit version of linked libraries?

> extSoftVersion()
                     zlib                     bzlib                        xz 
                  "1.2.8"      "1.0.6, 6-Sept-2010"              "5.1.0alpha" 
                     PCRE                       ICU                       TRE 
        "8.38 2015-11-23"                        "" "TRE 0.8.0 R_fixes (BSD)" 
                    iconv                  readline 
             "glibc 2.23"                     "6.3"

> library(sf)
Linking to GEOS 3.5.0, GDAL 2.1.2, proj.4 4.9.2
> sf::sf_extSoftVersion()
   GEOS    GDAL  proj.4 
"3.5.1" "2.1.2" "4.9.2" 

This information could be accessed by a funtion <pkgname_extSoftVersion>, see extSoftVersion and (sf_extSoftVersion()](https://github.com/edzer/sfr/blob/5c3dfea395af81bf352b4007d16c6a7d419883c2/R/init.R#L59)

• Linux tests via Travis (https://docs.travis-ci.com/user/languages/r#APT-packages)
• Windows tests via AppVeyor
• [ ] Mac OS X via Travis (http://ropensci.org/blog/2016/07/12/travis-osx) No Docker on Mac OS X: https://docs.travis-ci.com/user/reference/osx

Allow to manually select a specific R version

Now there are versioned Rocker images: https://github.com/rocker-org/rocker-versioned/

https://github.com/metacran/rversions might become handy.

• default is the "current" version of the generating machine, maybe use paste0(version$major, '.', version$minor)
• users can manually set an R version to use
• users can manually set a base image

https://github.com/cloudyr/googleComputeEngineR

How will/does/should containerit leverage other packages?

r-lib/devtools#710

we must add user config files etc. to the container and make sure they are actually used.

https://stat.ethz.ch/R-manual/R-devel/library/base/html/Startup.html

The startup script should ideally also log the environment, i.e.

• sessionInfo()
• https://github.com/metacran/rversions might be useful...

extends #6

The session loads rgdal and proj packages and adds the required libraries in the Dockerfile.

Use two approaches and then compare:

1. rsysreqs
2. https://github.com/rstudio/shinyapps-package-dependencies (in the Dockerfile: clone the repo then execute the respective install script)

In case information is missing, we can retrieve it from the user via a commandline interface directly from R

• based on package https://github.com/gaborcsardi/ask
• based on utils:menu, cf. https://github.com/hadley/devtools/blob/aaa4b61ca7c44515418d485cc64c84475e998ac7/R/utils.r#L69

When we can write Dockerfiles, we might as well parse them. uncontainer_it = create a session on a host machine that resembles the one that is inside a container.

Use cases needed!

- [ ] https://github.com/o2r-project/o2r-muncher/tree/master/test/bagtainers/markdowntainer-sfr/data

• https://github.com/o2r-project/o2r-muncher/tree/master/test/bagtainers/markdowntainer-units/data
• test for vignette sf3 from sf-package (available by system.file("doc/sf3.Rmd",package = "sf"))
• test for system.file("examples", "knitr-minimal.Rnw", package = "knitr")

Suggestion/discussion

We could wrap the Docker CLI in R functions.

• What is the advantage over using the CLI?

MRAN supports file style locations now: RevolutionAnalytics/checkpoint#218

Also, this issue RevolutionAnalytics/checkpoint#216 (comment) reports on how to make checkpoint work for an R Markdown document.

Extends #6 and #33

When reproducing an R session also match the locales.
As shown in #33 (comment) locales are not reproduced yet

In Linux, the locale first has to be generated, if missing, and then configured as default or current locale. That seems not to be trivial, especially in non-interactive mode.

The R functions Sys.getlocale() and Sys.setlocale() may be helpful.

• A sessionInfo can be reproduced with German locales (or anything other than the default)
• Testthat file tests/testthat/test_sessioninfo_reproduce.R runs with test (uncomment corresponding lines):

test_that("the locales are the same ", {
  expect_equal(local_sessionInfo$locale, docker_sessionInfo$locale)
})

Following the suggestions in Label Schema we can add some meta-information to the images

• metadata labels (generated when, by who, ..)
• arbitrary content (just create labels from a named list)
• put session info output into a label as plain text
• util function to read the labels from an image/container

Create a small session (load few packages, including one from CRAN but not in base packages) and create a Dockerfile that is as close as possible to recreate that session.

Based on sessionInfo()$running we have a mapping from running string to base image. In our case all running strings map to rocker.

Also check devtools::session_info(), could be useful to determine installation source.

Open questions:

• how can we fill the MAINTAINER field?

Consider how the function works_with_R could be useful for this package: https://github.com/tdhock/dotfiles/blob/master/.Rprofile

See also https://github.com/tdhock/works-poster

https://github.com/christophergandrud/repmis/blob/master/R/InstallOldPackages.R

NixOS is an interesting Linux distro that is all about declarative configuration and "devops", see e.g. https://www.domenkozar.com/2014/03/11/why-puppet-chef-ansible-arent-good-enough-and-we-can-do-better/

Maybe containerit could create NixOS installation instructions instead of Dockerfiles ?

• https://nixos.org/nixos/manual/index.html#ch-configuration
• https://github.com/zefhemel/nix-docker
• https://blog.wearewizards.io/why-docker-is-not-the-answer-to-reproducible-research-and-why-nix-may-be
• http://disq.us/p/1pfz951

When a user installs libs with pkgsnap then we should copy those libs over instead of downloading from CRAN.

Related to #30

See https://github.com/jimhester/lintr#testthat

• lint runs within a Travis Build

What will be the CMD or ENTRYPOINT?

• generate file
• write test

Maybe a helper script

They seems mostly reasonable :-)

https://github.com/ropensci/packaging_guide

We need a CLI (command line interface) wrapper around the library to integrate it into workflows in other programming languages (e.g. as part of a node.js-based webapp)

• implement based on docopt > https://github.com/docopt/docopt.R (see also this presentation)

Alternatively, if docopt does not work at all, evaluate package https://cran.r-project.org/web/packages/optparse/

Example usage:

container_it.R [-f <path to (markdown, R-script)file>]

container_it.R [-s] # package new R session

What options do we need exposed? How easy is that with docopt?

When packaging research into higher level containers, e.g. ERC, we most probably need some meta information. While the use can be asked for this, see #13, it would be better to extract this automagically from the session.

For this, we would need a feature that appends a script to the "main script file" of the container which has access to the R session "after" the analysis is completed.

Some ideas for informations that could be extracted here:

• temporal extend of objects (classes sptdf, xts, timeseries, ...)
• spatial extend of objects (classes sp, sf, raster, ...)
• user metadata? publication metadata?
  • parse markdown header
• names of (potential) "input" files and "output" files

This feature is complementary to the file analysis conducted by @7048730 in https://github.com/o2r-project/o2r-meta

Extends #6

Install a package that is only available from GitHub. Probably need devtools::session_info().

Packages for testing:

• https://github.com/edzer/sfr
• https://github.com/ropensci/geojsonio

This should (optinally, or even by default) also see if a specific version is tagged on GitHub and install that explicitly, e.g. devtools::install_github("Appsilon/shiny.collections", ref = "0.1.0")

https://rstudio.github.io/packrat/

Goal: add a snapshot of a packrat private repository to the container

https://cran.r-project.org/web/packages/datapack/vignettes/datapack-overview.html seems to have a good working process

• initial vignette works as part of regular R package build ("setup works")
• push modifications after review

see also #10

It could be useful to extend the Dockerfile so that the captured session is fully replicated directly after container start. This would save the user to call require/library on those packages manually.

The only way to restore an interactive session with required libraries seems to be defining a Rprofile.site file and setting R_PROFILE environment variable to its location (using ENV instruction)

The R_PROFILE file must contain a .First - function which attaches required packages using require(...) (or library)?

load namespaces via requireNameSpace() --> create instruction CMD ["R"] at the end of the Dockerfile

May extend #37

Users may include R objects from their R workspace in a restored session.
Therefore, the dockerfile()-method has a parameter 'objects' that is not yet implemented.

The objects-parameter takes a character vector containing the names of the objects, as returned by the function ls()

The objects are saved to an RData-file that is copied to the image at the location of the R working directory. If the file has no name (".RData"), R by default loads it automatically into the session on startup.

Alternatively, users may "load" the file manually from the working directory.

• implement objects-parameter save_image
• restore an arbitrary session with R objects
• write a test

Append all Dockerfiles used in FROM statements for a complete one-document Dockerfile. Could be interesting to evaluate for reproducibility - what is really installed?

"Unchain a Dockerfile"... _how useful can this be?

See e.g. Debian testing Dockerfile: https://github.com/tianon/docker-brew-debian/blob/9b1dd4b1594b8df02f7caa739e84b187edaab404/testing/Dockerfile

• 1. containerit executes the script locally and reproduce the session that results by the end of the script
• 2. copy = script (default) copies the supplied script to the image,
• 3. copy = script_dir also copies the script and all files / directories of the same folder
• 4. copy takes a list of files and directories to be copied to the folder
• 5. cmd parameter can be set with Cmd_Rscript("path/to/script") resulting to
  CMD ["Rscript", "--save", "/path/to/Rscript"]
• 6. Test 1-5 with test scripts

- [ ] 1. Default: execute a script locally and reproduce the session that results by the end of the script
- [ ] 2. copy_script = TRUE also copies the script
- [ ] 3. copy_parent = TRUE also copies the script and all files / directories of the same folder
- [ ] 4. batch_exec = TRUE also copies script and sets CMD instruction to
CMD ["Rscript", "--save", "/path/to/Rscript"]
- [ ] 5. copy_files takes a list of files and directories to be copied to the folder
- [ ] 6. Test 1-5 with test scripts

Issues (to be discussed later)

• How to handle: R working directory, relative/absolute paths, build context

Provide a path to a vignette, then it gets packaged as the main document in a container (build at start time of the container, container finished when vignette is build).

https://cran.rstudio.com/web/packages/switchr/

Base on https://docs.docker.com/engine/reference/builder/

See if stuff from traitecoevo/dockertest https://github.com/traitecoevo/dockertest/search?utf8=%E2%9C%93&q=dockerfile can just be reused?

It might be quite hard for long running system calls, but maybe we find a way to add a progress bar.

• use package https://github.com/gaborcsardi/progress or even colours with https://github.com/gaborcsardi/crayon

Use https://github.com/ropensci/datapack (see also https://cran.rstudio.com/web/packages/datapack/index.html)

Could be useful to install packages via packrat: https://github.com/rstudio/packrat

It is completely correct that right now there is an error message when a session is packaged, because containeRit itself is not published online: "Failed to identify source for package containeRit. Therefore the package cannot be installed in the docker image."

However, it should not be common to "package the packaging lib", so we should add an option add_self = false to the dockerfile(..) function that does by default not try to add the containeRit package itself to the image.

Simply copy over packages from lib directory / search path.

Throw errors if OS of host and container are not compatible.

Package a script and add, outside of the script, some result testing, i.e. a validation that the script has succeeded.

Probably use testthat for it.

RStudio has some nice ways to create a GUI (when package is used in RStudio presumably), so when needing user input this could be handy

See RStudio Add-inns: https://rstudio.github.io/rstudioaddins/

Issue #33 suggests to determine system dependencies based on https://github.com/rstudio/shinyapps-package-dependencies

However, as previous discussion in #33 shows, system dependencies are not explicit for all packages, for instance rgdal. That is because those dependencies that are listed in the basic Dockerfile seem to be pre-assumed, while the scripts from the 'packages' folder apply to packages that rely on additional dependencies. Hence, in order to rely on the shinyapps-package-dependencies, we would need to install all dependencies from the basic Dockerfile (or use the shinyapps image as the base image) which can result in unnecessary overhead.

Moreover, the shell scripts are made for ubuntu/linux only and therefore may not be applied to all potential base images.

• re-evaluate shinyapps-package-dependencies for re-use

Useful:

• gistr package

Candiates

• http://read.psych.uni-potsdam.de/index.php
• https://rpubs.com/edzer