elabftw / elabctl

elabctl is a tool to manage (install/backup/update) an eLabFTW installation

A free, modern, versatile, secure electronic lab notebook for researchers

Official website | Live demo | Documentation | Dedicated hosting

Description

eLabFTW is an electronic lab notebook manager for research teams.

It lets you store and organize your research experiments easily. It also features a database where any kind of objects (such as antibodies, plasmids, cell lines, boxes, etc.) can be stored. It is accessed via the browser. Several research teams can be hosted on the same installation. This means eLabFTW can be installed at the institute level and host all team members at the same place. This is what is done at in many research institutions around the globe. It is also a great solution for companies looking into improving the traceability and reproducibility of their processes.

Features

• Lab notebook for experiments
• Database for lab reagents, equipment, storage, cell lines, etc...
• Trusted timestamping
• Blockchain timestamping
• Import and export in various formats
• Calendar to manage booking of equipment
• Support for various scientific file formats
• Molecule/equation editor
• LaTeX support
• Todolist
• Public REST API
• Available in 17 languages
• Advanced permissions system
• Audited, secure codebase suitable for sensitive research results

How it works

eLabFTW is designed to be installed on a server, and people from the team can log in from their browser.

Installation

eLabFTW can easily be installed on any GNU/Linux server. It requires at least 512 MB of RAM (1 GB recommended) and 300 MB of disk space. Installation is done through Docker, so Docker needs to be installed on the server. Since it runs in a Docker container, php, webservers or mysql do not have to be installed on your server. Everything is provided by the containers so different versions and extensions do not matter. This makes upgrading versions much easier.

The documentation explains everything:

Installation instructions

Contributing

eLabFTW is a collaborative project. Your contributions are very welcome! Have a look at the contributing page to see how you can help with translation or code.

Security

See SECURITY.md.

Support

Community support

Free support is available through GitHub issues. Please open an issue if you have a problem, want to make a suggestion, ask a question or discuss a feature.

You can also join the chat room on Gitter.

Commercial support

PRO support and professional hosting are available from Deltablot.

Stay tuned

Subscribe to the newsletter to be notified when a new release is out!

Or use GitHub to follow releases:

Funding

eLabFTW is distributed without charge. If you find this project useful, please consider making a small donation either:

• as a GitHub Sponsor

• or with Liberapay

Gold sponsors

This project is backed by Deltablot company.

• Atelier de la donnée ADOC Lorraine (projet financé avec le soutien du Fonds national pour la Science ouverte)
• Department for the Development of Innovative Digital Use – French National Research Institute for Sustainable Development
• Institut Curie – Centre de Recherche
• Vrije Universiteit Brussel
• Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center – University of Freiburg, Germany

Related projects

API libraries

• elabftw/elabapi-python: python library for interacting with APIv2
• elabftw/elabapy: python library for interacting with APIv1 (deprecated)
• elabftw/elabapi-javascript-example: example code for interacting with APIv2 through JavaScript

Communities

• The ELN Consortium: a consortium of ELN vendors interested in improving interoperability between ELNs

Third party libraries or scripts

These projects were created by users of eLabFTW, check them out if interested!

• din14970/elabftwqrprint: python library to print QR codes from database objects
• shabihsherjeel/nionswift_elabftw_plugin: python plugin for Nionswift
• iam-cms/elabapy-cli: command line interface for eLabFTW API
• Elab::Client: perl module for eLab API by Andreas K. Hüttel
• LabView client: LabView client library by @plasmapper
• Matlab client: Matlab package to facilitate working with eLabFTW API
• elabftw-teamupload: Synchronize teamgroups and teams with users from Excel

Thank you for using eLabFTW. I hope it will bring happiness to your lab ;)

elabctl's Issues

check for disk free before update

see https://github.com/discourse/discourse_docker/blob/master/launcher#L206

create config.php

Due to installation woes, I tried the wget install method.
I am unfamiliar with mariadb, and nginx, and despite having run successfully, there are still database issues. Actualy, why doesnt the fresh drop create the appropriate config.php instead of a config.php-example? I mean, its setting up the user and database anyway, why not just have it create the appropraite configfile as part of the install?

anyhow, trying to use the wget drop was unsuccessfull. The log file seems to think it created a elabftw user, but my database says no such userexists.

mysqldump error after upgrade to elabctl v 2.3.3

When I do a backup via elabctl backup I get this error (but backup seems to proceed anyway):

mysqldump: [ERROR] unknown variable 'column-statistics=0'

Missing cleanup of TMP_DIR

Minor issue

elabctl creates a TMP_DIR here:

elabctl/elabctl.sh

Line 16 in 779b091

declare TMP_DIR=$(mktemp -d)

Since this is never removed a lot of dirs will be generated if you

1. leave the server running for a long time
2. say ;-) run elabctl in a daily cronjob for backups

debian@elabftw:~$ l /tmp/ | wc
93 738 5645

TMP_DIR itself is only used once in function self-update but the dependent variable TMP_CONF_FILE is often needed in function install

run backup without sudo

Dear Developper,
why i can not run the backup without sudo? can I change this behavior?

cheers
Luigi

Make elabctl update ask for (re)installing same version

Currently if you run elabctl update it pulls the docker images and restarts everything even when you already have the current version.

Would it be a better idea to look if latest version = installed version and offer to do nothing?
Are there possible downsides?
I tested something like this:

function update
{
    is-installed
    has-disk-space
    installed_release=$(docker exec -it ${ELAB_WEB_CONTAINER_NAME} grep INSTALLED_VERSION /elabftw/src/classes/App.php | cut -d\' -f 2)
    echo "Found installed release: $installed_release"
    latest_release=$(curl -s https://api.github.com/repos/elabftw/elabimg/releases/latest | jq -r ".tag_name")
    echo "Found latest release: $latest_release"
    if [ "$latest_release" = "$installed_release" ]; then
        echo "Notice: the latest intalled version appears to be the current version, do you want perform a reinstall? (y/N)"
        read -r reinstall_current
        if [ ! "$reinstall_current" = "y" ]; then
            echo "Not reinstalling current version!"
            exit 0
        fi
    fi
    echo "Do you want to make a backup before updating? (y/N)"
    ....
}

root@elabftw-dev:~# elabctl update
Using elabctl configuration file: /etc/elabctl.conf
Using elabftw configuration file: /etc/elabftw.yml
---------------------------------------------
Found installed release: 4.3.10
Found latest release: 4.3.10
Notice: the latest intalled version appears to be the current version, do you want perform a reinstall? (y/N)
n
Not reinstalling current version!

elabctl script not working on RHEL 7.3 (Maipo)

After manually installing some of the dependencies, I seem to have been able to get an installation of elabftw up and running, but the setup with elabctl does not work.

I will be happy to try and troubleshoot this, so the automated setup script will work

Add a cronjob to renew certs

.

Honoring /etc/elabctl.conf?

elabctl does not honor /etc/elabctl.conf, but it does honor the same file when it is present in the same directory as the executable. Not serious but has thrown us again off by assuming we can just have all the config files under /etc

Backup file sizes

My lab has now been using elabFTW for quite a while and it seems, that the size of the backup files does not increase in a linear fashion, but somehow exponentially (maybe its also that people use it differently and thus bigger backup files are produced). The present backup script is not very disk space friendly and I have forked the elabctl project and have added a snapshot command to elabctl that uses rsync to make incremental, uncompressed backups. After the first backup (which is larger than a compressed backup), any incremental backup uses much less disk space and is also is faster. However, the biggest issue is the experiments_revisions table, which is still need to address since the incrementals that I use are at the file level (and once can only mysqldump whole tables, but not deltas). I am going to test my first release soon on our production system (we started to get complaints from the IT guys that our backups fill the hard drive or the VM).

install page broken using this script

i just tried following these instructions: http://elabftw.readthedocs.io/en/master/install-drop.html

i end up with a blank install page :

i've tried 3 times now on two different servers - so i'm pretty sure there is a bug in the script.

the only noticeable discrepancy between the instructions and my actions is that the 1-click app Docker image on DO is now 1.11.2

elabctl entries for docker-compose lacking required "-"

When doing a fresh install of current image (From web page: ,Installed version: 4.2.0-BETA Latest version: 4.1.0),
"elabctl start" fails at line that should invoke docker-compose. Since the "-" is missing in the elabctl script, that cmd
is interpreted as a docker command and fails with a cmd line syntax error. Editing the downloaded elabctl script to
insert the proper "-" in 4 places corrects the problem and elabftw installs properly.

Not sufficiently familiar with git to send off a proper patch, but the fix is pretty obvious and only shows up in four
lines. Verified that latest elabctl github version shows the above error.

Otherwise, latest version seems to be working properly.......great app.

Regards,
rtisys

Self-signed certificate request ignored by installer script?

Hi,
I am just moving our elabftw install from ubuntu 18.04 to 20.04. Before doing that, I test-installed (for curiosity) a fresh version of elabftw on the newly created virtual server and always ran into trouble since elabctl status always showed the elabftw container as "Up (unhealthy)". I repeated the process a few times since I do not trust myself. But finally, I inspected the generated /etc/elabftw.yml file and realized that ENABLE_LETSENCRYPT=true. I had not set up Letsencrypt since I wanted to use self-signed certificates and had chosen that option during the installation. After manually setting ENABLE_LETSENCRYPT=false, everything worked perfectly. So obviously choosing to use self-signed certificates is not honored by the installer script. I repeated the install (deleting the existing /etc/elabftw.yml file) and again, my request was not honored. Even though I fully agree with you to get a "real" certificate, there are plenty of use cases where self-signed certs are needed, e.g. test installations. And since our elabftw install is only available inside our university's firewall, I need to get the certs from our university (and install them manually), which is usually the last thing to do after everything else works...

elabctl backup doesn't honor non-standard locations for uploaded files

Our university's server farm offers variable and expandable server space via a separate /data block device. Hence I need to put the uploaded files somewhere under that directory. The corresponding section of our elabftw.yml file looks like this:

    volumes:
        # this is where you will keep the uploaded files persistently
        # host:container
        - /data/elabftw/web:/elabftw/uploads

However, it seems that the "elabctl backup" command doesn't backup the files in my directory at all, and also doesn't generate an error message, because the /var/elabftw/web directory exists in my case, but is unused, since the files are uploaded to /data/elabftw/web. In order to use the elabftw backup, it should use the volumes defined in /etc/elabftw.yml or accept a command line argument to change the default behavior.

Issue with older versions of Docker and elabctl backup

It looks like some older versions of docker have an issue when elabctl backup is run under cron.

Description: If elabctl backup is executed from a tty the script executes normally, but fails when executed by cron.

In particular this line (in elabctl) causes an error due to the tty call.
docker exec -it mysql bash -c 'mysqldump -u$MYSQL_USER -p$MYSQL_PASSWORD -r dump.sql $MYSQL_DATABASE' > /dev/null 2>&1

It can be fixed by changing the line to the one below.
docker exec -i mysql bash -c 'mysqldump -u$MYSQL_USER -p$MYSQL_PASSWORD -r dump.sql $MYSQL_DATABASE' > /dev/null 2>&1

This issue MAY be fixed in the latest versions of docker but I have been unable to verify this. However this might be helpful to someone out there.

Docker version 1.13.0, build 49bf474
Ubuntu 16.04.1 LTS (Running on AWS)
elabctl version 0.2.1

Program 'jq' seems to be a prerequisite - mention this in install instructions?

Hi,

I ran into a small issue running elabctl update on Debian:

I got "Notice: 'jq' command not found, skipping release is beta check."

I did not have jq installed. Perhaps jq might be mentioned as a prerequisite in the installation instructions?

Best,
Christian

Intended behaviour of the update mechanism

I was looking at a backup solution for eLabFTW machines, and stumbled upon a somewhat surprising bit in the elabctl update and backup semantics. In order of discovery:

• The current version of the script copies an elabftw.yml file so that a container can be restored (to whatever has the right tags online).
• When restoring a previously working system from a backup, it is nice if the data and the code are consistent. The SQL dump might not be consistent with just any version of the code.
• This does appear to be the responsibility of the user. The default is to use elabftw/elabimg:latest, and if we use this, the saved data will need some massaging to get it to run, if there have been eg schema changes or additional settings should be set (either to get it to run, or to get it to run securely and reliably). There are instructions about to how to change this.
• I propose that this is streamlined. I started a minor update in #26
• My code does not change the backed-up elabftw.yml, since this is not necessarily consistent with the elabctl update semantics that some might expect. (And note the number of "expect"-like words. This is just about consistency of expectations, and ease-of-use.)
• The current behaviour och elabctl update is to update to the latest versions of the containers. This is what is stated, and of course true regardless of which containers are pointed out (so, technically correct - the best kind of correct). But based on the update instructions on the web page, the intention seems to be to update elabftw, rather than just go from an old version of the container tagged 4.0.1 to the latest bugfixed version of the container tagged 4.0.1 (as these things in practice seem to take place mostly between version numbers).
• One idea would be to have the elabftw.yml file reflect the current numbered version (eg 4.2.4, rather than "latest") and have the update mechanism update this.
• This would require the elabctl script to somehow figure out the latest version (the hacky solution would be "change to latest, pull, rewrite" but it's not nice).
• Additionally, if there are large changes over and above just running update, of the run update2to3 variety, it would be nice if it suggests or enforces such updates.

There are of course multiple things that I haven't thought of, and I don't know the paradigmatic way of implementing it. Thus the issue, rather than me trying to write a fully-fledged hack ex ante. :)

issues with automated letsencrypt

i just tried the automated install on a couple of fresh Ubuntu 14.04.4 x64 droplets, with both versions of the instructions. i thought i'd document my findings to help make this even smoother/easier for others.

i could not get this script to work (from latest):

wget -qO- https://get.elabftw.net|sh

this script (from next) did eventually work for me, with the caveats below:

wget -qO- https://get.elabftw.net > install-elabftw.sh && sh install-elabftw.sh

my letsencrypt failed a few times because i had a very small terminal window, this is a known issue: certbot/certbot#2787

i only succeeded in getting the 'standalone' version (option 2 in the letsencrypt dialog) working. the webroot version failed for me, but perhaps i messed that up. if someone can get that working, more explicit instructions would be helpful.

it might also be helpful if the script was able to check for a successful letsencrypt install. if that fails, all of the elabftw install messages point to success - which is slightly confusing. not sure how involved that would be to implement.

this script is a really powerful tool - i'm really glad you are working to making elabftw even easier to install!

Backup only runs when elabctl is started

I use elabctl v0.6.3.

This will most likely require a bit more information, which I'll be happy to provide when asked.

I wanted to update my elabftw to the most recent version, and I ssh'ed into the server, ran
sudo elabctl stop
sudo elabctl update

It returned the following, and immediately returned to the prompt, seemingly without doing anything;

__@elabftw01:~$ sudo elabctl update
elabctl © 2017 Nicolas CARPi - https://www.elabftw.net
Version: 0.6.3
Using configuration file: /etc/elabftw.yml

Before updating, a backup will be created.
Using backup directory /var/backups/elabftw
__@elabftw01:~$ ls /var/backups/elabftw/

I checked the backup directory, and it did not create any files.
I then started the server again (sudo elabctl start), and reran the update. This time it did not return to the prompt immediately, but did in fact generate the backup files, and also carried out the update.

After restarting the services, I am met with only an error message;
SQLSTATE[HY000] [2002] Connection refused
Or a similar message regarding a timeout.

I restarted the server it is running on, and after restarting the service, I can monitor the status with elabctl status, consecutively:

__@elabftw01:~$ sudo elabctl status
elabctl © 2017 Nicolas CARPi - https://www.elabftw.net
Version: 0.6.3
Using configuration file: /etc/elabftw.yml

 Name               Command               State             Ports         
-------------------------------------------------------------------------
elabftw   /run.sh                       Up           0.0.0.0:443->443/tcp 
mysql     docker-entrypoint.sh mysqld   Restarting                        
__@elabftw01:~$ sudo elabctl status
elabctl © 2017 Nicolas CARPi - https://www.elabftw.net
Version: 0.6.3
Using configuration file: /etc/elabftw.yml

 Name               Command             State          Ports         
--------------------------------------------------------------------
elabftw   /run.sh                       Up      0.0.0.0:443->443/tcp 
mysql     docker-entrypoint.sh mysqld   Up      3306/tcp, 33060/tcp  
__@elabftw01:~$ sudo elabctl status
elabctl © 2017 Nicolas CARPi - https://www.elabftw.net
Version: 0.6.3
Using configuration file: /etc/elabftw.yml

 Name               Command               State             Ports         
-------------------------------------------------------------------------
elabftw   /run.sh                       Up           0.0.0.0:443->443/tcp 
mysql     docker-entrypoint.sh mysqld   Restarting                        
__@elabftw01:~$ sudo elabctl status
elabctl © 2017 Nicolas CARPi - https://www.elabftw.net
Version: 0.6.3
Using configuration file: /etc/elabftw.yml

 Name               Command             State          Ports         
--------------------------------------------------------------------
elabftw   /run.sh                       Up      0.0.0.0:443->443/tcp 
mysql     docker-entrypoint.sh mysqld   Up      3306/tcp, 33060/tcp  
__@elabftw01:~$ sudo elabctl status
elabctl © 2017 Nicolas CARPi - https://www.elabftw.net
Version: 0.6.3
Using configuration file: /etc/elabftw.yml

 Name               Command               State             Ports         
-------------------------------------------------------------------------
elabftw   /run.sh                       Up           0.0.0.0:443->443/tcp 
mysql     docker-entrypoint.sh mysqld   Restarting                        
__@elabftw01:~$ sudo elabctl status
elabctl © 2017 Nicolas CARPi - https://www.elabftw.net
Version: 0.6.3
Using configuration file: /etc/elabftw.yml

 Name               Command               State             Ports         
-------------------------------------------------------------------------
elabftw   /run.sh                       Up           0.0.0.0:443->443/tcp 
mysql     docker-entrypoint.sh mysqld   Restarting   

Note the mysql; In the beginning it toggles between Up and Restarting, but then it settles on Restarting.

This is a bit beyond my skill-set - what should I try next?

error during update (docker install)

My elabftw is down, here is the error I have during update:

# elabctl update     

      _          _     _____ _______        __
  ___| |    __ _| |__ |  ___|_   _\ \      / /
 / _ \ |   / _| | '_ \| |_    | |  \ \ /\ / / 
|  __/ |__| (_| | |_) |  _|   | |   \ V  V /  
 \___|_____\__,_|_.__/|_|     |_|    \_/\_/   
                                              

Using elabctl configuration file: using default values (no config file found)
Using elabftw configuration file: /etc/elabftw.yml
---------------------------------------------
Do you want to make a backup before updating? (y/N)

Pulling mysql ... done
Pulling web   ... done
Removing mysql ... done
Creating mysql ... 

ERROR: for mysql  a bytes-like object is required, not 'str'

ERROR: for mysql  a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.21/containers/94d139d8f91d1eabeafd8d316f4e279acff2b8293c2f6548eea35db5235da8e0/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/compose/service.py", line 625, in start_container
    container.start()
  File "/usr/lib/python3/dist-packages/compose/container.py", line 241, in start
    return self.client.start(self.id, **options)
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/docker/api/container.py", line 1095, in start
    self._raise_for_status(res)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'cgroups: cgroup mountpoint does not exist: unknown'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 11, in <module>
    load_entry_point('docker-compose==1.25.0', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 72, in main
    command()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 128, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1107, in up
    to_attach = up(False)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1088, in up
    return self.project.up(
  File "/usr/lib/python3/dist-packages/compose/project.py", line 565, in up
    results, errors = parallel.parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/project.py", line 548, in do
    return service.execute_convergence_plan(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 545, in execute_convergence_plan
    return self._execute_convergence_create(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 460, in _execute_convergence_create
    containers, errors = parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 465, in <lambda>
    lambda service_name: create_and_start(self, service_name.number),
  File "/usr/lib/python3/dist-packages/compose/service.py", line 457, in create_and_start
    self.start_container(container)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 627, in start_container
    if "driver failed programming external connectivity" in ex.explanation:
TypeError: a bytes-like object is required, not 'str'
Your are now running the latest eLabFTW version.
Make sure to read the CHANGELOG!
=> https://github.com/elabftw/elabftw/releases/latest

Please, help

Install fails on lets encrypt

I'm trying to install elab on a server.
I've made a new user and add the user to sudo and gluster groups.
I run elabctl, and edit the config to point to a directory in the gluster mount.
I run elabctl install, and accept all defaults.
The installer runs and exits with the following

Requesting to rerun ./letsencrypt-auto with root privileges...
Requested domain  is not a FQDN because it contains an empty label.

When I look for the files/dirs in the config, all I see is data. Which has letsencrypt mysql web in it.

Error updating (image not found)

When updating elabftw using elabctl from a digitalocean droplet server:

Status: Image is up to date for mysql:5.7
Pulling web (elabftw/docker-elabftw:latest)...
Pulling repository docker.io/elabftw/docker-elabftw
ERROR: Error: image elabftw/docker-elabftw:latest not found

The rest of the update proceeds and elabctl start reports that elabftw is up-to-date, but the sysadmin panel reports out-of-date.

borg-backup fails if BORG_REMOTE_PATH is not set

elabctl 3.6.1
bash 4.2.46

If BORG_REMOTE_PATH is undefined in elabctl.conf, elabctl borg-backup fails with this error:
/usr/local/bin/elabctl: line 58: BORG_REMOTE_PATH: unbound variable

It is because of set -eu on line 54.

Suggested fix: replace block at line 58

if [ -n "$BORG_REMOTE_PATH" ]; then
   export BORG_REMOTE_PATH="${BORG_REMOTE_PATH}"
fi

with

if [[ -v BORG_REMOTE_PATH ]]; then
   export BORG_REMOTE_PATH="${BORG_REMOTE_PATH}"
fi

It might be obvious but I'd like to stress nevertheless: it's -v BORG_REMOTE_PATH and not -v "$BORG_REMOTE_PATH"

don't require dialog and have a fallback

.

DUMP_DELETE_DAYS missing as config in elabctl.conf

The new feature "Delete old mysql dumps" in 3.4.0 is missing the corresponding option in elabctl.conf.

Also there should be a way to disable it by setting this to 0.