aiidalab / aiidalab-docker-stack

Docker images with the basic software stack for AiiDAlab

Docker Stack for AiiDAlab

This repository contains the Dockerfiles for the official AiiDAlab docker image stack. All images are based on the jupyter/minimal-notebook.

Image variants:

• base – A minimal image that comes with AiiDA pre-installed and an AiiDA profile set up.
• base-with-services – Like base, but AiiDA services (PostgreSQL and RabbitMQ) are installed on the container and automatically launched on startup.
• lab – Like base, but uses the AiiDAlab home app as the primary interface (the standard JupyterLab interface is also available).
• full-stack – Our most comprehensive image, like lab, but also comes with services pre-installed and launched.

Supported tags (released on Docker Hub):

• edge – the latest commit on the default branch (main)
• latest – the latest regular release
• aiida-$AIIDA_VERSION – the latest regular release with that AiiDA version (ex. aiida-2.0.0)
• python-$PYTHON_VERSION – the latest regular release with that Python version (ex. python-3.9.2)
• $version – the version of a specific release (ex. 2022.1001)

In addition, edge, latest, and $version are also released internally on the GitHub Container registry (ghcr.io). Pull requests into the default branch are further released on ghcr.io with the pr-### tag to simplify the testing of development versions.

Quickstart

You can launch a container based on one of our published images directly with Docker, by executing for example the following command:

docker run -it -p 8888:8888 aiidalab/full-stack

However, we recommend to use AiiDAlab Launch to run images locally for production environments.

Note: On recent versions of Mac OS-X you will have to select a different port, since port 8888 is already in use by one of the system services.

Known limitations

• Resetting the username and thus home directory location from the default (jovyan) via the NB_USER environment variable is currently not supported (#297).

Development

Development environment

The repository uses the doit automation tool to automate tasks related to this repository, including building, testing, and locally deploying Docker images with docker-compose.

To use this system, setup a build end testing environment and install the dependencies with:

pip install -r requirements-dev.txt

Build images locally

To build the images, run doit build (tested with docker buildx version v0.8.2).

The build system will attempt to detect the local architecture and automatically build images for it (tested with amd64 and arm64). All commands build, tests, and up will use the locally detected platform and use a version tag based on the state of the local git repository. However, you can also specify a custom platform or version with the --platform and --version parameters, example: doit build --arch=arm64 --version=my-version.

You can specify target stacks to build with --target, example: doit build --target base --target full-stack.

Run automated tests

To run tests, first build the images as described in the previous section. Then run the automated tests with doit tests.

Tip: The continuous integration workflow will build, release (at ghcr.io/aiidalab/*:pr-###), and test images for all pull requests into the default branch.

For manual testing, you can start the images with doit up, however we recommend to use aiidalab-launch to setup a production-ready local deployment.

Continuous integration

Images are built for linux/amd64 and linux/arm64 during continuous integration for all pull requests into the default branch and pushed to the GitHub Container Registry (ghcr.io) with tags ghcr.io/aiidalab/*:pr-###. You can run automated or manual tests against those images by specifying the registry and version for both the up and tests commands, example: doit up --registry=ghcr.io/ --version=pr-123. Note: You may have to log into the registry first.

Creating a release

We distinguish between regular releases and special releases, where the former follow the standard versioning scheme (v2022.1001) and the latter would be specific to a certain use case, e.g., a workshop with dedicated requirements. To create a regular release, set up a development environment, and then use bumpver:

bumpver update

This will update the README.md file, make a commit, tag it, and then push both to the repository to kick off the build and release flow.

To create a special release, simply tag it with a tag name of your choice with the exception that it cannot start with the character v.

Deploy AiiDAlab with AiiDAlab Launch

The aiidalab-launch tool provides a convenient and robust method of both launching and managing one or multiple AiiDAlab instances on your computer. To use it, simply install it via pip

pip install aiidalab-launch

and then start AiiDAlab with

aiidalab-launch start

Note: AiiDAlab will keep running until you explicitly stop it or shutdown/restart your computer. In that case, you will have to run the aiidalab-launch start command again to restart AiiDAlab.

Please see aiidalab-launch --help for a full list of available commands and options.

Cloud and other deployments

Please see the AiiDAlab documentation for information on how to use and deploy AiiDAlab docker images in alternative ways.

Citation

Users of AiiDAlab are kindly asked to cite the following publication in their own work:

A. V. Yakutovich et al., Comp. Mat. Sci. 188, 110165 (2021). DOI:10.1016/j.commatsci.2020.110165

Acknowledgements

This work is supported by the MARVEL National Centre for Competency in Research funded by the Swiss National Science Foundation, as well as by the MaX European Centre of Excellence funded by the Horizon 2020 EINFRA-5 program, Grant No. 676598.

aiidalab-docker-stack's Issues

Update rdkit and openbabel verisons

To have a more robust implementation of the SMILES converter
conversion should be first tried via rdkit via a try---except
and then via openbabel.

For some strange reason the present versions of openbabel and rdkit interfer one each other
it should be better to install:

rdkit=2020.03.6
openbabel=3.1.1

https://github.com/aiidalab/aiidalab-widgets-base/blob/cce7876ab87b3b5876683c4a71c62dfc3117fbd9/aiidalab_widgets_base/structures.py#L517

a correct installation should allow to process the following sequence:
`from openbabel import pybel as pb
from openbabel import openbabel as ob
from rdkit.Chem import MolFromSmiles,MolToMolFile
from rdkit import Chem
from rdkit.Chem import AllChem
#some pybel code
#end some pybel code

m = Chem.MolFromSmiles("C")
m = Chem.AddHs(m)

AllChem.EmbedMolecule(m, maxAttempts=20, randomSeed=42)
AllChem.UFFOptimizeMolecule(m,maxIters=100)
m.GetConformer().GetPositions()`

the following sequence instead (with present way to upload pubel) crashes:

`import pybel as pb
import openbabel as ob
from rdkit.Chem import MolFromSmiles,MolToMolFile
from rdkit import Chem
from rdkit.Chem import AllChem
#some pybel code
#end some pybel code

m = Chem.MolFromSmiles("C")
m = Chem.AddHs(m)

AllChem.EmbedMolecule(m, maxAttempts=20, randomSeed=42)
AllChem.UFFOptimizeMolecule(m,maxIters=100)
m.GetConformer().GetPositions()`

Import SSSP from existing export files

http://archive.materialscloud.org/file/2018.0001/v1/SSSP_efficiency_pseudos.aiida
http://archive.materialscloud.org/file/2018.0001/v1/SSSP_accuracy_pseudos.aiida

Hopefully this will go some way towards speeding up this part
https://github.com/materialscloud-org/mc-docker-stack/blob/37437fd16de6a433e55cfd51239b6d8fd4d9d242/start-singleuser.sh#L76-L82

Setup PYTHONPATH through Docker arguments (or similar)

For the moment PYTHONPATH is defined through the .bashrc file which does not always get executed. For example, if one runs aiida calculations through the submit function the PYTHONPATH definition in .bashrc is not taken into account.

Failure of aiidalab/aiidalab-test-app-action@v2 action

When tests are run, the aiidalab-test-app-action always fails with the following output:

Options:
  --version          Show version number                               [boolean]
  --app-path, -a     Path to the app's directory.  [string] [required] [default:
                "/home/runner/work/aiidalab-docker-stack/aiidalab-docker-stack"]
  --image, -i        The aiidalab image to test on.
                         [default: "aiidalab/aiidalab-docker-stack:sha-51619d6"]
  --name, -n         The name of the app within the apps folder.
                                                               [default: "home"]
  --bundled          Assume that the app is pre-installed and test that one
                     using the tests in the provided app-path.
                                                       [boolean] [default: true]
  --browser          Specify which browser to use.
                     [choices: "chrome", "firefox", "opera"] [default: "chrome"]
  --screenshots, -s  Path to bind the screenshots volume to.  [string] [default:
    "/home/runner/work/aiidalab-docker-stack/aiidalab-docker-stack/screenshots"]
  --notebooks        Which notebooks to test with generic tests.
                                                     [default: "**/[!.]*.ipynb"]
  --help             Show help                                         [boolean]

Missing required argument: app-path
Please set the app-path either directly with -a/-app-path=path/to/app or via the $GITHUB_WORKSPACE environment variable.

The image is built with tornado==6.1

Currently the container is erroneously built with tornado==6.1 albeit only being compatible with tornado<5. This affects all recent images.

To reproduce, for example with the recently tagged release (v20.11.0):

git checkout v20.11.0
docker build . -t aiidalab-docker-stack:v20.11.0 --no-cache  # --no-cache is important!!
docker run --name=aiidalab aiidalab-docker-stack:v20.11.0

Then enter the container with

$ docker exec -it aiidalab /bin/bash
# su aiida
$ pip list --format=freeze | grep tornado
tornado==6.1

This is a critical issue, because it affects all images (even older ones) and renders the image unusable for submissions, e.g., with the QE app.

deprecated log_file setting in jupyterhub config

reported by @yakutovicha

The setting "c.JupyterHub.extra_log_file" in the jupyterhub_config.py is now deprecated. Developers suggest to use the redirection of the output. To do so one needs to figure out how to redirect the output of the system service.

install and configure cp2k and QE codes

Those two codes are required by the example app to work properly

make new release

once the 1.0 container is somewhat converged, please make a new release - both of the repo and on dockerhub so that the AiiDA lab can start pointing to fixed versions.

Add more ENV variables

That is in sync with aiidalab/aiidalab#31:

We need to add variables such as AIIDALAB_DEFAULT_GIT_BRANCH to specify which default branch should be used when installing an app. Also, it might be useful to have AIIDALAB_DEFAULT_APPS to provide a list of apps to be installed by default.

issues when building locally

Hi,
I was trying to build the develop branch locally (to check if adding the bigdft plugin was working). I encountered few issues when launching the container.

• md5 command used in run_aiidalab.sh is not available on my ubuntu system, it seems to be osx specific ? md5sum is the one available, so I had to replace line 15 by
  TOKEN=date | md5sum | cut -c -32

• 404 error (from jupyter, so the server runs well) in browser when launching the generated container , it says it can't find http://localhost:8888/apps/apps/home/start.ipynb . Launching the container from docker hub works, though, so I don't know if it's an issue with local builds only or my changes

The only thing I change in the recipe is using pip install -e to install aiidalab package from a repository instead of pip, so the diff is basically

+COPY opt/aiidalab /opt/aiidalab
+RUN pip install -e /opt/aiidalab
+
 RUN pip install                    \
-    'aiidalab==v20.03.0b2'         \

with opt/aiidalab being a clone of https://github.com/aiidalab/aiidalab with just the aiida-bigdft plugin added.

Edit: and I confirm that undoing all the changes (using the dockerfile from this repo) does not fix things, I still get a 404.

Any idea where this could come from ?

Opening up for non-root execution of the stack

Due to the rather extensive set of platforms now not supporting root executions in containers we should consider to opening up for regular users to execute the stack.

The services we require for aiidalab can be executed as non-root, but there has been questions regarding the my_init script and the framework beneath of the phusion baseimage that we use can support it. We would like to continue using this baseimage due to its process tracking and reaping.

It turns out this is possible, but we loose cron and logrotate functionality. Do we need this? Paging @yakutovicha @csadorf. I do not think so. Maybe the latter is a problem, but if the logfile becomes so big that it truncates, the systems that requires non-root usually have HA in place, which restarts the container anyway and then you can analyse what is wrong given the current status. We should document this clearly though. See for instance posts: phusion/baseimage-docker#475, especially: phusion/passenger-docker#250 (comment). With build_args we can disable cron.

Enable automated migration of the PostgreSQL database.

This issue has the following dependencies:

• aiidateam/aiida-prerequisites#41
• aiidalab/aiidalab-home#94

---

The migration scripts down below are based on the official migration recipes as suggested by @ltalirz.

To clarify:

• Identify at which point of the container's life the migration should happen: at startup or at runtime?
  [Answer]: It should be run at runtime as we cannot guarantee that there is enough time during the startup.
• Decide whether we want to enable the migration inside the aiidalab (or aiida) container, or maybe we want to provide a separate dedicated container for this?
  [Answer]: It will be installed in the same container.
  • Should we install both: the old and new versions of Postgres in the same container. [Answer: YES]
  • Depending on the previous answer: how would we interface migration to the user? Should we run it in the background, or should we put a big message on the startup screen saying that "Image is updated, migration is required"?
    [Answer]: In AiiDAlab, it would be a part of the Home app. The user will be notified about the possible migration and provided with an option to run it. The migration will block all the AiiDA operations.
• Agree whether we want to use the recommended PostgreSQL way of migration (e.g. via pg_dump) or should we use less tested pg_upgrade instead?
  [Answer: pg_dump]

Possible upgrade path:

1. Have both (previous and current) versions of PostgreSQL in the same container
2. In case the default version is not the newest one - signal to the user about this: "New version of PostgreSQL is available, please upgrade".
3. Block the interface for the whole upgrade process.
4. Perform the dump of the database from the old PostgreSQL.
5. Switch to the newer PostgreSQL and import the dump.
6. Switch the default PostgreSQL
7. Restart the container.
8. Signal to the user that the old database and database dump are still in the filesystem. Propose them to delete those.

migrate SSSP export archives

Currently, the SSSP export archives are migrated whenever a new container is started (and this takes time)

Just store the already migrated archives in the image.

List of hindrances when starting to develop

When starting to develop after running:

./run.sh 8888 ~/aiidalab

I run into a list of hindrances before I can really get to work. I'll keep track of them here. Not sure if these can be "fixed", but the steps to resolve them an get to work should be documented. Or a different pathway to starting to develop should be explained or easy to find.

• Although the apps are installed in the ~/aiidalab directory and hence easily accessible, you still have to change the ownership to your user before you can start adapting them or using e.g. git commands:

  (qeapp) mbercx@theospc46:~/aiidalab$ ls -l
  total 4
  drwxrwxr-x 5 administrator administrator 4096 Jun  5 09:49 apps
  

  Once I run:

  sudo chown -R mbercx:administrator apps/
  

  I'm in business! Maybe we can change the ownership in the setup?
• For future releases that ship with the aiidalab-qe-workchain: immediately install this package in editable mode. This is really just for yours truly, but would be nice if I could skip the step to have to log in as root and do this. 😁
• Although upon initial deployment, the link+token is outputted so I can get to work right away, when I restart my computer for example I need to give the token again when I log in. If I hadn't kept this somewhere, I wouldn't really know how to proceed, except for restart the deployment.
• The docker container now has a default generated name like strange_dewdney or heuristic_wiles. Is there a reason not to specify this name?

PS: I thought there were more when I first started opening this issue, but seems I just messed up a lot when I last set up this dev environment. 😅 Will still add anything I can think of to make the dev process smoother here.

try to avoid copying conda activation scripts from .bashrc

The /opt/prepare-aiidalab.sh script currently copies the conda init lines from the .bashrc .

It would be nice to find a way not to have to do this (we tried sourcing .bashrc but it failed, likely because of this)

enable %aiida magic function

for the inspiration check out here and here

should containers work outside JupyterHub

@espenfl notes that

with the aiida one I get JUPYTERHUB_API_TOKEN env is required to run jupyterhub-singleuser.

Perhaps it would make sense if one could run the aiidalab docker container also outside the JupyterHub?

apt-get install fails with Undetermined Error

While building the Docker image locally (EPFL) works fine, building the same docker image at CSCS OpenStack results in an error

Get:202 http://archive.ubuntu.com/ubuntu bionic/main amd64 libffi-dev amd64 3.2.1-8 [156 kB]
Fetched 133 MB in 1min 23s (1589 kB/s)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/g/gcc-7/libasan4_7.3.0-16ubuntu3_amd64.deb  Undetermined Error [IP: 91.189.88.152 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

run.sh failed without error message

I tried running the following on MacOS 11.5.2 with Docker Desktop 3.6.0

./run.sh --no-build --image aiidalab/aiidalab-docker-stack:latest 8888 /Users/leopold/Personal/Postdoc-MARVEL/repos/aiidalab/aiidalab-docker-stack/vol
Launching the container...
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Waiting for container to start...

The script returned without giving me a link to a container or an error message.

The container did not start; I don't know why

$ docker logs 4bbe7e27a9bd
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/10_create-system-user.sh...
/usr/bin/id: ‘aiida’: no such user
no change     /opt/conda/condabin/conda
no change     /opt/conda/bin/conda
no change     /opt/conda/bin/conda-env
no change     /opt/conda/bin/activate
no change     /opt/conda/bin/deactivate
no change     /opt/conda/etc/profile.d/conda.sh
no change     /opt/conda/etc/fish/conf.d/conda.fish
no change     /opt/conda/shell/condabin/Conda.psm1
no change     /opt/conda/shell/condabin/conda-hook.ps1
no change     /opt/conda/lib/python3.7/site-packages/xontrib/conda.xsh
no change     /opt/conda/etc/profile.d/conda.csh
no change     /home/aiida/.bashrc
No action taken.
*** Running /etc/my_init.d/10_syslog-ng.init...
Sep 13 10:37:25 4bbe7e27a9bd syslog-ng[268]: syslog-ng starting up; version='3.13.2'
*** Running /etc/my_init.d/20_start-rabbitmq.sh...
 * Starting RabbitMQ Messaging Server rabbitmq-server
   ...done.
*** Running /etc/my_init.d/30_start-postgres.sh...
pg_ctl: no server running
waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
*** /etc/my_init.d/30_start-postgres.sh failed with status 1

*** Killing all processes...
Sep 13 10:37:43 4bbe7e27a9bd syslog-ng[268]: syslog-ng shutting down; version='3.13.2'

@yakutovicha any hints?

move to sqlalchemy backend

Cannot install with pip in editable mode

After trying to install a locally git cloned repository using pip install -e ., it failed with a message of writing rights to the file system.

Note, it installs fine if not installing as editable.

To reproduce, open a terminal and do the following (preferrably under the apps folder).

$ git clone https://github.com/unkcpz/aiidalab-sssp-workflow
$ pip install -e aiidalab-sssp-workflow

In other words, setup.py develop doesn't work, but setup.py install works fine.

deployment docs both in README and in aiidalab.readthedocs.io

The docs on how to deploy the AiiDAlab container currently are present both in the README of this repo and on
https://aiidalab.readthedocs.io/en/latest/deployment/index.html

I will add a link from the README to the aiidalab docs to remove this duplication

mentioning @csadorf

can we remove tini?

https://github.com/materialscloud-org/mc-docker-stack/blob/37437fd16de6a433e55cfd51239b6d8fd4d9d242/Dockerfile#L122-L127

See https://github.com/krallin/tini#using-tini

Inadvertent installation of outdated version of optimade-client

We are explicitly installing the aiidalab-optimade package here:

aiidalab-docker-stack/Dockerfile

Lines 96 to 99 in 08496a4

	# Install OPTIMADE.
	WORKDIR /opt/
	RUN git clone https://github.com/aiidalab/aiidalab-optimade.git && cd aiidalab-optimade && git reset --hard v1.2.1
	RUN pip install -e ./aiidalab-optimade

This caused a problem in the latest release, because we failed to update this dependency, which means that in the final image we inadvertently installed an outdated version of the optimade-client through the aiidalab-optimade dependency:

$ pip list --format=freeze | grep optimade
aiidalab-optimade==1.2.1
optimade==0.12.4
optimade-client==2020.10.29

We could of course update this, but since we are no longer relying on aiidalab-optimade in aiidalab-widgets-base, could we instead remove this completely?

consider switching to jupyter docker stack as a base image

There are lots of nice docker stacks here. We might be able to save ourselves quite a bit of maintenance by switching to one of those as our base image.

Note: The hierarchy is:
ubuntu 18.04 => base notebook => minimal notebook => scipy notebook => data science notebook

add symlink from /project to /home/user for backwards compatibility

In order not to break e.g. gotocomputer commands for old data, simply add a symlink /project that points to /home/user

Use a proper init script from phusion

See why here: https://github.com/phusion/baseimage-docker#contents

For services already provided by phusion, check e.g. https://github.com/aiidateam/torquessh_base-docker/blob/17731c2452187adb03661683ba9e25b11b079946/Dockerfile#L36 (note that what one needs to do differs depending on the base image from Phusion).

This to provide more init scripts:
https://github.com/aiidateam/torquessh_base-docker/blob/17731c2452187adb03661683ba9e25b11b079946/Dockerfile#L50

and CMD should always remain CMD ["/sbin/my_init"]

Move to conda environment.

For inspiration, one can look into Jupyter Docker Stacks and, more specifically, in the corresponding Docker file.

Closed apps' kernel keeps running

The right behaviour should be stopping the kernel when closing the corresponding tab. Currently, this does not happen.

aiida daemon fails to start on k8s

The daemon does not start on k8s using the current develop branch (using aiida-core 0.12.3).

For some reason, celery on k8s thinks it is being asked to run as root:
The daemon log file shows:

Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!

If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).

User information: uid=1000 euid=1000 gid=0 egid=0

This information is incorrect - e.g. id -g scientist yields group id 1000, not 0.
I wonder where celery gets this information from.

You can make the daemon run by telling celery it's ok to run as root using export C_FORCE_ROOT=1 but I'd rather have celery get the correct information.

SSSP_efficiency_v1.0 family missing

for some reason, the SSSP_efficiency_v1.0 pseudo family was missing in my aiida profile.
Reason: unknown

verdi setup fails for aiida-1.0 image on k8s

Using the aiida-1.0-k8s branch.

This is the end of the logs of the pod

+ verdi setup --profile default --non-interactive --email [email protected] --first-name Some --last-name Body --institution XYZ --db-backend django --db-username aiida --db-password aiida_db_passwd --db-name aiidadb --db-host localhost --db-port 5432 --repository /project/aiida_repository
Success: created new profile `default`.
Info: migrating the database.
Critical: database migration failed, probably because connection details are incorrect:
could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Cannot assign requested address
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?

Any ideas why it isn't able to connect to postgres?

Note: This problem does not occur on the develop branch.

upcoming migration of materials cloud archive

this is just to notify @yakutovicha that the migration of the materials cloud archive to invenio is currently planned for May 4th.

while old links to records should be redirected, it is possible that the same won't be true for files associated with records.

e.g. the link to the SSSP export archives might change to something like

https://archive.materialscloud.org/record/file?file_id=b5e3eb47-7151-43c3-8f4b-f8a50789f152&filename=SSSP_precision_pseudos.tar.gz&record_id=22

container startup fails if `localhost` not configured in default profile

The startup of containers fails, if the default AiiDA profile does not have a localhost computer configured:

+ verdi code show cp2k@localhost
Usage: verdi code show [OPTIONS] CODE

Error: Invalid value for 'CODE': no Code found with LABEL<cp2k@localhost>: No result was found
++ which cp2k.popt
+ verdi code setup --non-interactive --label cp2k --description 'cp2k v5.1 in AiiDA lab container.' --input-plugin cp2k --computer localhost --remote-abs-path /usr/bin/cp2k.popt
Usage: verdi code setup [OPTIONS]

Error: Invalid value for '-Y' / '--computer': no Computer found with LABEL<localhost>: No result was found
*** /etc/my_init.d/80_prepare-aiidalab.sh failed with status 2

cheat sheet refers to management of AiiDA lab server

the "cheat sheet" section of the README actually lists commands for managing a docker-based AiiDA lab multi-user server (rather than a cheat sheet for commands to use inside the docker container).

https://github.com/aiidalab/aiidalab-docker-stack#cheat-sheet

I propose to move this cheat sheet here:
https://github.com/aiidalab/ansible-playbook-aiidalab-server/blob/master/README.md

CI: Switch to build-push-action v2 (supports caching)

The CI workflows currently take up to 20 mins, because we build the docker image each time anew (~ 15mins). This step could be skipped in the vast majority of cases if we would take advantage of the GitHub actions caching mechanisms which are supported by the build-push-action v2.

Replace default user info in aiida profile

https://github.com/ltalirz/mc-docker-stack/blob/37437fd16de6a433e55cfd51239b6d8fd4d9d242/start-singleuser.sh#L50-L55

One could use the JUPYTERHUB_USER environment variable (holds user email).
Perhaps one needs to add more environment variables for user name & affiliation?
We just need to forward this information from the auth server.

This is important as soon as any of the data generated in jupyter are to be exported.

Figure out why pip install overwrites upper-level dependencies

To be verified.

If one runs pip install package the expected behaviour is to install locally the dependencies that are missed in the system. Instead, pypi installs all the dependencies locally independently if they were installed before.

add docker badge using microbadger

https://microbadger.com/

looks very useful, see e.g. https://pypi.org/project/sslyze/

Base aiidalab-docker-stack on aiidateam/aiida-core container

To simplify testing and give a more clear separation of concerns.

installing postgresql twice

@yakutovicha You're currently installing postgresql twice in the aiidalab-1.0-migration branch:

version 10

aiidalab-docker-stack/Dockerfile

Line 41 in 530c7f5

postgresql \

and version 9.6

aiidalab-docker-stack/Dockerfile

Lines 72 to 77 in 530c7f5

	# Add repo for postgres-9.6
	RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main" >> /etc/apt/sources.list.d/pgdg.list
	RUN wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | apt-key add -
	RUN apt-get update && apt-get install -y --no-install-recommends \
	postgresql-9.6 \
	&& rm -rf /var/lib/apt/lists/*

What ends up running is version 10:

# ps -ef | grep postgresql
scienti+    43     1  0 19:59 ?        00:00:00 /usr/lib/postgresql/10/bin/postgres -D /project/.postgresql

This will bite you when you try to migrate databases from the previous image that was running postgres 9.6.
Either you add some migration scripts or you switch back to version 9.6

Install openbabel in the container.

This is the code that might potentially work:

RUN apt-get update
RUN apt-get install swig cmake -y --no-install-recommends
ENV LD_LIBRARY_PATH="/usr/local/lib/"
RUN git clone https://github.com/openbabel/openbabel.git
RUN mkdir openbabel/build
RUN cd openbabel/build && cmake .. -DPYTHON_BINDINGS=ON -DRUN_SWIG=ON
RUN cd openbabel/build && make
RUN cd openbabel/build && make install


RUN apt-get install pkg-config
RUN cp -r openbabel/scripts/python/openbabel /usr/local/lib/python3.6/dist-packages/
RUN cp -r openbabel/build/lib/_openbabel.so /usr/local/lib/python3.6/dist-packages/openbabel

Travis: aiida daemon is not running

At the end of Travis run the aiida daemon was not running.

I don't understand why this is so, maybe the same happens in the aiida-core base image.

delete aiida_1.0_migration branch

the aiidalab_server role can install both from dockerhub and from the repo.

In order to keep things simple, I've now copied the aiida_1.0_migration into an aiida-1.0 branch on the repo to match the aiida-1.0 docker tag (I've also connected the docker tag to it).

@yakutovicha I.e. once you've updated the openstack roles, the aiida_1.0_migration branch can be deleted.

Delete jupyterlab branch?

I am not sure this branch is needed, and it is quite outdated by now.

add .local/bin to path

By default, ~/.local/bin is not in the PATH, and therefore executables installed via pip install --user cannot be found

add default apps

• AiiDA tutorials
• CSCS computers
• calculation examples

(should appear in this order)

Possibility to deploy a clean AiiDAlab

Right now, e.g. QE is installed by default and that seems rather hard coded. We should open up for an easy way to avoid this and just deploy the basics of AiiDAlab.

In my view this would also not include material science related widgets, only the computer, code etc.

Remove ipytree from Pipfile

ipytree will be specified as the dependency of AWB (aiidalab/aiidalab-widgets-base#184), so it should be removed from the Pipfile:

aiidalab-docker-stack/Pipfile

Line 38 in dedcce1

ipytree = "~=0.1.8" # required by qe-app

Feature request: Factory reset

Desired feature

It should be possible for administrators and users to initiate a factory reset, that basically returns the state of the user environment to "factory conditions".

Motivation

• This would make it easier for users to restore a working system state after rendering it inadvertently unusable through manual package installations.
• It would allow administrators and tutorial leaders to test user workflows under factory conditions.

Additional notes

• A factory reset could be automatically recommended if it is detected that the environment was not accessed for a long time (e.g. multiple months).
• It might be possible to configure whether the reset only resets the software environment or wipes user data as well.
• The factory reset function could be implemented as part of the home app or as a separate app.