e4s-project / e4s-cl

Container manager for E4S

E4S Container Launcher

E4S container launcher is a tool to simplify running MPI applications in E4S containers with host libraries. This project is built upon the MPICH ABI Compatibility Initiative and CEA's Wi4MPI.

Check out the documentation here !

Usage

Running an application with E4S Container Launcher can be as simple as prefixing the launcher command by e4s-cl launch.

e4s-cl init
e4s-cl launch mpirun -np 4 ./foo

The full details on execution and configuration can be found here.

How it works

Creating containers

e4s-cl wraps around standard MPI-capable launchers to execute a small amount of pre-processing before creating containers in which to run commands. This processing includes analyzing the library environment from the host, to bind select libraries to the container in order to make them accessible to the running process.

The concept of profile

One of the core components of the tool are profiles. They contain specific information about an execution environment that can be reproduced, such as container image and technology, or files and libraries to import in the containers.

They can be accessed and modified using the e4s-cl profile commands.

$ e4s-cl profile list
============= Profile Configurations (/home/user/.local/e4s_cl/user.json) ==
+----------+------------+-------------+------------+-----------+-------+--------+
| Selected |    Name    |   Backend   |   Image    | Libraries | Files | WI4MPI |
+==========+============+=============+============+===========+=======+========+
|          |  MPICH_3.4 | singularity |       None |    16     |   1   |   No   |
+----------+------------+-------------+------------+-----------+-------+--------+

Profiles are also used as intermediate outputs of the program. e4s-cl profile detect will analyze a given binary to list all its library and file dependencies. Those results are stored into a given profile to be reused.

Installation

e4s-cl requires python3.7+ to run. If you do not have access to a compatible version, the guided installation process will download a conda interpreter to run the project.

Guided standalone installation

You can install e4s-cl and a dedicated python interpreter using the provided Makefile. The installation location can be set using the INSTALLDIR variable:

git clone https://github.com/E4S-Project/e4s-cl
cd e4s-cl
make INSTALLDIR=<prefix> install

Once the installation succeeds, add <prefix>/bin to your PATH.

python package installation

If you would rather install e4s-cl using standard python packaging facilities, you may install using pip:

git clone https://github.com/E4S-Project/e4s-cl
cd e4s-cl
pip install .

The full details on installation targets can be found here.

e4s-cl's Issues

Missing mpirun arguments

Detection of jsm_pmix on Summit

Something is not detected correctly on Summit, in the /opt/ibm/spectrum_mpi/jsm_pmix/ tree.

Issue detecting a profile on single node system

I am trying to set profile detect to set up my profile. Here is my sample Fortran program that sums the ranks (run with 11 ranks isum = 55):

PROGRAM hello_world_mpi
include 'mpif.h'

integer process_Rank, size_Of_Cluster, ierror
integer root_rank, isum

call MPI_INIT(ierror)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size_Of_Cluster, ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD, process_Rank, ierror)


root_rank = 0
call  MPI_Reduce(process_rank, isum, 1, MPI_INT, MPI_SUM, root_rank, MPI_COMM_WORLD, ierror);
call MPI_bcast (isum, 1, MPI_INTEGER, root_rank, MPI_COMM_WORLD, ierror)

print *, 'Hello World from process: ', process_Rank, 'of ', size_Of_Cluster, 'sum = ', isum
end program

I compiled it with mpiifort

$ mpiifort -v
mpiifort for the Intel(R) MPI Library 2019 Update 9 for Linux*
Copyright 2003-2020, Intel Corporation.
ifort version 19.1.3.304

This program runs with mpirun

$ mpirun --version
Intel(R) MPI Library for Linux* OS, Version 2019 Update 9 Build 20200923 (id: abd58e492)
Copyright 2003-2020, Intel Corporation.
$ mpirun -np 11  -hosts lscamd50-d.gfdl.noaa.gov ./test.x

Here is my e4s-cl command

$ e4s-cl profile detect -p am4Run mpirun -np 11  -hosts lscamd50-d.gfdl.noaa.gov ./test.x
Failed to determine necessary libraries.

The advice in the documentation is to specify multiple hosts (https://e4s-project.github.io/e4s-cl/reference/profiles/detect.html#profile-detect), but this is a single node system with 128 cores. How can I get all of the libraries needed to run on my system?

`podman` default options prevent container analysis

Passing --env=host to the podman process can overwrite the container's $PATH, and if the libc version is old enough to not have generators in the LD cache, the analysis fails to find the required ldconfig binary.

The result is a defensive avoid of library import that result in the wrong libc version to be used.

RO filesystems exception

On RO filesystems, the logging fails and cripples the program. Should be fixed, as some batch schedulers land you in a RO filesystem (Like jsrun on summit).

Initialization failure with mpi libraries omitting 'mpirun'

The e4s-cl init script fails when a librarie does not embark mpirun in its binaries.

$ ./e4s-cl init --mpi /usr/tce/packages/mvapich2/mvapich2-2.3-intel-19.0.4
No MPI detected in PATH. Please load a module or use `--mpi` to specify the MPI installation to use.

Fine-grain library filtering

When the libc versions do not match, and the container's library is more recent, a filter is applied and every library present in the container is omitted from the import list.
On Illyad, this process fails, as libibverbs.so.1 is present in both the host and container. However, it is way more recent on the host, despite the lower version of the libc.

By looking at the library's symbols before omission, those issues could be prevented.

Executable not found during launch

When launching a container using e4s-cl to run an executable, command not found error is thrown:

$ e4s-cl launch --profile init_runsi  mpirun -np 4 a.out
38860 on cyclops: /.e4s-cl/script: line 18: a.out: command not found

But giving the absolute path for the executable makes it run without issues:

$ e4s-cl launch --profile init_runsi  mpirun -np 4 $PWD/a.out
We have 4 processes.
Process 1 reporting for duty.
Process 2 reporting for duty.
Process 3 reporting for duty.

It looks like singularity (the container technology used with this profile) doesn't always keep the user's working directory within the launched container.

e4s-cl init crash on crusher

Crash when using stock init on crusher.

[[email protected] ~]$ e4s-cl -v init
[Debug e4s_cl.cli.commands.__main__:195] Verbosity level: DEBUG
[Debug e4s_cl.cf.storage.local_file:50] '/autofs/nccs-svm1_home1/jskutnik/SHARED/E4SCL_INSTALLATION/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/autofs/nccs-svm1_home1/jskutnik/SHARED/E4SCL_INSTALLATION/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/autofs/nccs-svm1_home1/jskutnik/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/autofs/nccs-svm1_home1/jskutnik/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(cmd=[])
[Debug root:25] Failed to retrieve generator: no extensions in cache
[Debug root:79] find library=libmpi.so.12; searching
[Debug root:63] search path=/sw/aaims/crusher/build/smartbench/lib:/opt/cray/pe/papi/6.0.0.15/lib64:/opt/cray/libfabric/1.15.0.0/lib64		(LD_LIBRARY_PATH)
[Debug root:67] trying file=/sw/aaims/crusher/build/smartbench/lib/libmpi.so.12
[Debug root:67] trying file=/opt/cray/pe/papi/6.0.0.15/lib64/libmpi.so.12
[Debug root:67] trying file=/opt/cray/libfabric/1.15.0.0/lib64/libmpi.so.12
[Debug root:94] search cache=/etc/ld.so.cache
[Debug root:63] search path=/lib:/usr/lib:/lib64:/usr/lib64		(SYSTEM)
[Debug root:67] trying file=/lib/libmpi.so.12
[Debug root:67] trying file=/usr/lib/libmpi.so.12
[Debug root:67] trying file=/lib64/libmpi.so.12
[Debug root:67] trying file=/usr/lib64/libmpi.so.12
[Debug root:79] find library=libmpi.so.40; searching
[Debug root:63] search path=/sw/aaims/crusher/build/smartbench/lib:/opt/cray/pe/papi/6.0.0.15/lib64:/opt/cray/libfabric/1.15.0.0/lib64		(LD_LIBRARY_PATH)
[Debug root:67] trying file=/sw/aaims/crusher/build/smartbench/lib/libmpi.so.40
[Debug root:67] trying file=/opt/cray/pe/papi/6.0.0.15/lib64/libmpi.so.40
[Debug root:67] trying file=/opt/cray/libfabric/1.15.0.0/lib64/libmpi.so.40
[Debug root:94] search cache=/etc/ld.so.cache
[Debug root:63] search path=/lib:/usr/lib:/lib64:/usr/lib64		(SYSTEM)
[Debug root:67] trying file=/lib/libmpi.so.40
[Debug root:67] trying file=/usr/lib/libmpi.so.40
[Debug root:67] trying file=/lib64/libmpi.so.40
[Debug root:67] trying file=/usr/lib64/libmpi.so.40
[Debug e4s_cl.cli.commands.init:187] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
No MPI compiler detected. Please load a module or use the `--mpi` option to specify the MPI installation to use.
[Debug e4s_cl.error:168] Traceback (most recent call last):
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/bin/e4s-cl", line 12, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/__main__.py", line 199, in main
    return cli.execute_command([cmd], cmd_args)
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/lib/python3.9/site-packages/e4s_cl/cli/__init__.py", line 314, in execute_command
    return find_command(cmd).main(cmd_args)
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/init.py", line 469, in main
    status = _analyze_binary(args)
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/init.py", line 203, in _analyze_binary
    returncode = detect_command.main(command)
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/profile/detect.py", line 268, in main
    args = self._parse_args(argv)
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/lib/python3.9/site-packages/e4s_cl/cli/command.py", line 76, in _parse_args
    args = self.parser.parse_args(args=argv)
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/lib/python3.9/argparse.py", line 1824, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/ccs/home/jskutnik/SHARED/E4SCL_INSTALLATION/conda/lib/python3.9/argparse.py", line 1836, in parse_known_args
    args = list(args)
TypeError: 'int' object is not iterable

Statically compiled shared objects mistaken as linkers

MPICH 3.4.2 uses a library containing only data. This library has thus no dynamic dependencies and is understood as a linker by e4s-cl.

Error 'multiple repeat in regex string'

Running with mvapich2:

quartz2123{user1}51: e4s-cl mpirun -np 4 ./ring
Using selected profile default-4d58938472ef20d7983858a9769c80e3225ba6af4212de66264756adec8dcf6d
Inside container
Inside container
Inside container
Inside container
36567 on quartz2123: An unexpected error exception was raised:

multiple repeat at position 8

Traceback (most recent call last):
  File "/usr/global/tools/tau/training/e4s-cl-July21/bin/e4s-cl", line 17, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/usr/global/tools/tau/training/e4s-cl-July21/bin/../packages/e4s_cl/cli/commands/__main__.py", line 204, in main
    return cli.execute_command([cmd], cmd_args)
  File "/usr/global/tools/tau/training/e4s-cl-July21/bin/../packages/e4s_cl/cli/__init__.py", line 320, in execute_command
    return find_command(cmd).main(cmd_args)
  File "/usr/global/tools/tau/training/e4s-cl-July21/bin/../packages/e4s_cl/cli/commands/__execute.py", line 207, in main
    for line in libset.ldd_format():
  File "/usr/global/tools/tau/training/e4s-cl-July21/bin/../packages/e4s_cl/cf/libraries/libraryset.py", line 360, in ldd_format
    return list(map(line, set.union(self.sonames, self.missing_libraries)))
  File "/usr/global/tools/tau/training/e4s-cl-July21/bin/../packages/e4s_cl/cf/libraries/libraryset.py", line 352, in line
    if lib := self.find(soname):
  File "/usr/global/tools/tau/training/e4s-cl-July21/bin/../packages/e4s_cl/cf/libraries/libraryset.py", line 310, in find
    matches = set(filter(lambda x: re.match(soname, x.soname), self))
  File "/usr/global/tools/tau/training/e4s-cl-July21/bin/../packages/e4s_cl/cf/libraries/libraryset.py", line 310, in <lambda>
    matches = set(filter(lambda x: re.match(soname, x.soname), self))
  File "/usr/global/tools/tau/training/e4s-cl-July21/conda/lib/python3.8/re.py", line 191, in match
    return _compile(pattern, flags).match(string)
  File "/usr/global/tools/tau/training/e4s-cl-July21/conda/lib/python3.8/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/global/tools/tau/training/e4s-cl-July21/conda/lib/python3.8/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/global/tools/tau/training/e4s-cl-July21/conda/lib/python3.8/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/global/tools/tau/training/e4s-cl-July21/conda/lib/python3.8/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/global/tools/tau/training/e4s-cl-July21/conda/lib/python3.8/sre_parse.py", line 671, in _parse
    raise source.error("multiple repeat",
re.error: multiple repeat at position 8

This is a bug in E4S Container Launcher.
Please raise an issue on Github with the contents of '/g/g24/user1/.local/e4s_cl/debug_log’.

Support initialization scripts on a per-profile basis

Zoltan needs to be run using:

e4s-cl launch <launcher> bash -i -c ./run.sh

With run.sh containing:

#!/bin/bash

source `spack location -i lmod`/lmod/lmod/init/bash
module load gcc/7.3.0-yzxt
spack load -r trilinos
export LD_LIBRARY_PATH=/hostlibs:$LD_LIBRARY_PATH

./Zoltan/Zoltan

Ideally, this should be handled by the following:

e4s-cl launch <launcher> ./Zoltan

Profile showing fails when pattern matches multiple

The special case where a profile name is a pattern identifying another profile prevents it from being shown.

[jskutnik@spoutnik e4s-cl]$ e4s-cl profile list test
== Profile Configurations (/home/jskutnik/.local/e4s_cl/user.json) =============

+----------+--------+-------------+------------------------+-----------+-------+
| Selected |  Name  |   Backend   |         Image          | Libraries | Files |
+==========+========+=============+========================+===========+=======+
|          |   test | singularity | /home/jskutnik/wdm.sim |     0     |   0   |
|          |        |             |                      g |           |       |
+----------+--------+-------------+------------------------+-----------+-------+
|          | test02 |         N/A |                  /test |     0     |   0   |
+----------+--------+-------------+------------------------+-----------+-------+
|          | test03 |         N/A |                    N/A |     0     |   1   |
+----------+--------+-------------+------------------------+-----------+-------+
|          | test01 |         N/A |                  /test |     0     |   0   |
+----------+--------+-------------+------------------------+-----------+-------+

[jskutnik@spoutnik e4s-cl]$ e4s-cl profile show test
[Error] usage: e4s-cl profile show [arguments] <profile_name>
[Error] e4s-cl profile show: error: Pattern test does not identify an existing profile

Duplicate mount destination error when launching container

Hi,

I've been running into an issue where e4s-cl fails to launch a container because it tries to bind multiple host libraries to the same path inside the container.

e4s-cl launch --backend=podman --image docker-archive:mpibench.tar --profile=t4 srun -N 1 /mpiBench
[+] Using selected profile t4
srun: job 2351 queued and waiting for resources
srun: job 2351 has been allocated resources
Error: /.e4s-cl/hostlibs/libibverbs.so: duplicate mount destination
Process 961819 failed with code 125
Error: /.e4s-cl/hostlibs/libibverbs.so: duplicate mount destination
Container command failed with error code 125

In the above, the container runtime complains that /usr/lib64/libibverbs.so and /lib64/libibverbs.so can't both be bound to the same destination. On the system I'm working on, /lib64 is a symlink to /usr/lib64 and we have a few libraries that use RPATH with those two paths interchangeably. It seems e4s-cl is somehow picking up the same library at both locations.

e4s-cl/e4s_cl/cf/containers/__init__.py

Line 153 in 843cbe9

return bound_rhs.origin.relative_to(

The workaround we came up with is to patch the function in e4s-cl that checks for duplicate binds to instead only check if bound_rhs.destination == bound_lhs.destination.

Currently the duplicate detection only triggers when both the source and destination paths are identical and thus fails to detect the case where two source paths would end up at the same location inside the container.

Exception snowball on container failure

When the container backend command fails, a few exceptions pile up in a indigestible error message: error.log

Wi4MPI installation when unnecessary

Wi4MPI is installed at every init command, while being sometimes superfluous. This can result in an error message when there is no issue.

Apptainer/singularity: File does not exist in container

On some configurations, the container backend (apptainer/singularity) refuses to run with the following error:

FATAL:   container creation failed: 
mount /usr/lib/openmpi/libompi_dbg_msgq.so->/usr/lib/openmpi/libompi_dbg_msgq.so error: 
while mounting /usr/lib/openmpi/libompi_dbg_msgq.so: 
destination /usr/lib/openmpi/libompi_dbg_msgq.so doesn't exist in container

This is due to a file binding order/duplicate. In the above example, both /usr and /usr/lib/openmpi/libompi_dbg_msgq.so are bound to the container, and the backend mixes things up as one overwrites the other.

Uncaught exception when module not found / setup.py failure to list correct dependencies

[jskutnik@spoutnik e4s-cl]$ pip install .
Defaulting to user installation because normal site-packages is not writeable
Processing /home/jskutnik/Projects/e4s-cl
Requirement already satisfied: python-ptrace>=0.9.7 in /home/jskutnik/.local/lib/python3.10/site-packages (from e4s-cl==1.0rc6) (0.9.8)
Requirement already satisfied: pyelftools==0.27 in /home/jskutnik/.local/lib/python3.10/site-packages (from e4s-cl==1.0rc6) (0.27)
Requirement already satisfied: requests==2.26.0 in /home/jskutnik/.local/lib/python3.10/site-packages (from e4s-cl==1.0rc6) (2.26.0)
Requirement already satisfied: tinydb==4.5.2 in /home/jskutnik/.local/lib/python3.10/site-packages (from e4s-cl==1.0rc6) (4.5.2)
Requirement already satisfied: python-sotools==0.0.3 in /home/jskutnik/.local/lib/python3.10/site-packages (from e4s-cl==1.0rc6) (0.0.3)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.10/site-packages (from requests==2.26.0->e4s-cl==1.0rc6) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /home/jskutnik/.local/lib/python3.10/site-packages (from requests==2.26.0->e4s-cl==1.0rc6) (2022.5.18.1)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/jskutnik/.local/lib/python3.10/site-packages (from requests==2.26.0->e4s-cl==1.0rc6) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.10/site-packages (from requests==2.26.0->e4s-cl==1.0rc6) (3.3)
Using legacy 'setup.py install' for e4s-cl, since package 'wheel' is not installed.
Installing collected packages: e4s-cl
    Running setup.py install for e4s-cl ... done
Successfully installed e4s-cl-1.0rc6
[jskutnik@spoutnik e4s-cl]$ e4s-cl profile list
An unexpected ModuleNotFoundError exception was raised:

No module named 'docker'

Traceback (most recent call last):
  File "/home/jskutnik/.local/bin/e4s-cl", line 5, in <module>
    from e4s_cl.cli.commands.__main__ import COMMAND as cli_main_cmd
  File "/home/jskutnik/.local/lib/python3.10/site-packages/e4s_cl/cli/commands/__main__.py", line 13, in <module>
    from e4s_cl.cli.commands.launch import COMMAND as LAUNCH_COMMAND
  File "/home/jskutnik/.local/lib/python3.10/site-packages/e4s_cl/cli/commands/launch.py", line 52, in <module>
    from e4s_cl.cf.containers import EXPOSED_BACKENDS
  File "/home/jskutnik/.local/lib/python3.10/site-packages/e4s_cl/cf/containers/__init__.py", line 370, in <module>
    import_module(_module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/jskutnik/.local/lib/python3.10/site-packages/e4s_cl/cf/containers/docker.py", line 7, in <module>
    import docker
ModuleNotFoundError: No module named 'docker'

This is a bug in E4S Container Launcher.
Please raise an issue on Github with the contents of '/home/jskutnik/.local/e4s_cl/logs/debug_log'.

Singularity: Failure to bind certain files: destination does not exist in container

Using Singularity 3.8.5. The error message states the following:

FATAL:   container creation failed: mount /uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/openmpi-4.1.1-k6tvmwyoqqicgdmpve56jk37ktchjuyz/etc/pmix-mca-params.conf->/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/openmpi-4.1.1-k6tvmwyoqqicgdmpve56jk37ktchjuyz/etc/pmix-mca-params.conf error: while mounting /uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/openmpi-4.1.1-k6tvmwyoqqicgdmpve56jk37ktchjuyz/etc/pmix-mca-params.conf: destination /uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/openmpi-4.1.1-k6tvmwyoqqicgdmpve56jk37ktchjuyz/etc/pmix-mca-params.conf doesn't exist in container

SINGULARITY_BIND contains the following:

SINGULARITY_BIND=/uufs/chpc.utah.edu/common/home/u0101881:/uufs/chpc.utah.edu/common/home/u0101881:rw,/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/openmpi-4.1.1-k6tvmwyoqqicgdmpve56jk37ktchjuyz/etc/pmix-mca-params.conf:/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/openmpi-4.1.1-k6tvmwyoqqicgdmpve56jk37ktchjuyz/etc/pmix-mca-params.conf:rw,/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/openmpi-4.1.1-k6tvmwyoqqicgdmpve56jk37ktchjuyz/etc/openmpi-mca-params.conf:/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/openmpi-4.1.1-k6tvmwyoqqicgdmpve56jk37ktchjuyz/etc/openmpi-mca-params.conf:rw,/etc/libnl/classid:/etc/libnl/classid:rw

The file in question is neither a symlink nor behind a symlinked directory.

Crash when image is not compatible with host architecture

[jskutnik@cyclops ~]$ e4s-cl launch mpirun -np 2 ./MPI_TESTS/bw_test/bw
147259 on cyclops: FATAL:   could not open image /home/users/jskutnik/mpich.simg: the image's architecture (amd64) could not run on the host's (ppc64le)
147258 on cyclops: FATAL:   could not open image /home/users/jskutnik/mpich.simg: the image's architecture (amd64) could not run on the host's (ppc64le)
147259 on cyclops: An unexpected InternalError exception was raised:

Could not determine container's content !

Traceback (most recent call last):
  File "/home/users/jskutnik/.local/bin/e4s-cl", line 17, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/storage/users/jskutnik/.local/bin/../packages/e4s_cl/cli/commands/__main__.py", line 106, in main
    return cli.execute_command([cmd], cmd_args)
  File "/storage/users/jskutnik/.local/bin/../packages/e4s_cl/cli/__init__.py", line 300, in execute_command
    return main(cmd_args)
  File "/storage/users/jskutnik/.local/bin/../packages/e4s_cl/cli/commands/execute.py", line 251, in main
    container.get_data(params, library_set=libset)
  File "/storage/users/jskutnik/.local/bin/../packages/e4s_cl/cf/containers/__init__.py", line 138, in get_data
    raise InternalError("Could not determine container's content !")
e4s_cl.error.InternalError


This is a bug in E4S Container Launcher.
Please raise an issue on Github with the contents of '/storage/users/jskutnik/.local/e4s_cl/debug_log'.
147258 on cyclops: An unexpected InternalError exception was raised:

Could not determine container's content !

Traceback (most recent call last):
  File "/home/users/jskutnik/.local/bin/e4s-cl", line 17, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/storage/users/jskutnik/.local/bin/../packages/e4s_cl/cli/commands/__main__.py", line 106, in main
    return cli.execute_command([cmd], cmd_args)
  File "/storage/users/jskutnik/.local/bin/../packages/e4s_cl/cli/__init__.py", line 300, in execute_command
    return main(cmd_args)
  File "/storage/users/jskutnik/.local/bin/../packages/e4s_cl/cli/commands/execute.py", line 251, in main
    container.get_data(params, library_set=libset)
  File "/storage/users/jskutnik/.local/bin/../packages/e4s_cl/cf/containers/__init__.py", line 138, in get_data
    raise InternalError("Could not determine container's content !")
e4s_cl.error.InternalError


This is a bug in E4S Container Launcher.
Please raise an issue on Github with the contents of '/storage/users/jskutnik/.local/e4s_cl/debug_log'.

Completion is slow and can fail with errors

The current completion script loads the entire project to complete very simple options. A C++ script can reduce the latency and error rate of completion.

Error message when WI4MPI is not in use

The code will send out an error message when WI4MPI is not detected, even if that does not impair execution.

Singularity: error with PATH commands

Running a command in e4s-cl with no path (e.g. e4s-cl ls) fails when the guest glibc is being overrun by the host's. The error message is the following:

ls: error while loading shared libraries: ls: cannot open shared object file

This does not happen when using a relative or absolute path, and may be due to a process parsing the PATH loading the container's glibc and thus creating conflict when the host glibc is loaded.

Handle GLIBC differences graciously

Details here.

Compilation fails with IBM compilers

Compiling from stdin gcc style is not supported with Spectrum MPI's compiler. This induces fatal errors in e4s-cl init.

Bug report related to e4s-cl init

I am running on a cloud cluster set up on a parallel works platform. I was trying to run e4s-cl init, but I ran into an error, and after looking at documentation, I set up a profile

e4s-cl profile detect -p pw srun --mpi=pmi2 -N 3 -n 9  ./c_hello.x 

Then I tried to run e4s-cl with my profile and got an error saying I should report a bug here

$ e4s-cl init --launcher srun --backend singularity --profile pw
icc: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider running `e4s-cl profile detect` using mpirun specifying multiple hosts.
Failed to list opened files of /tmp/tmpgwhuuqtz: [Errno 2] No such file or directory
srun: error: thomasrobinson-fv3-00009-1-0007: task 0: Exited with exit code 156
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
An unexpected KeyError exception was raised:

2

Traceback (most recent call last):
  File "/contrib/bin/e4s-cl", line 12, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/__main__.py", line 200, in main
    return cli.execute_command([cmd], cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/__init__.py", line 314, in execute_command
    return find_command(cmd).main(cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/init.py", line 436, in main
    controller.update({'name': requested_name}, profile.eid)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/model/profile.py", line 102, in update
    super().update(data, keys)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/mvc/controller.py", line 236, in update
    database.update(data, keys, table_name=self.model.name)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cf/storage/local_file.py", line 459, in update
    table.update(fields, doc_ids=[keys])
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 340, in update
    self._update_table(updater)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 709, in _update_table
    updater(table)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 337, in updater
    perform_update(table, doc_id)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 326, in perform_update
    table[doc_id].update(fields)
KeyError: 2

This is a bug in E4S Container Launcher.
Please raise an issue on Github with the contents of '/home/Thomas.Robinson/.local/e4s_cl/logs/debug_log'.

Here's the contents of the file

$ cat /home/Thomas.Robinson/.local/e4s_cl/logs/debug_log
[Debug root:491] Unlink /home/Thomas.Robinson/.local/e4s_cl/logs/latest failed: [Errno 2] No such file or directory: '/home/Thomas.Robinson/.local/e4s_cl/logs/latest'
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:21:03.378903
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : fd02aa3ff3e94da26c65add3747862b90677a8053e9498c33c7214f7cf2e804b
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cf.storage.local_file:146] Initialized system filesystem prefix '/contrib/system'
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmpkv6f6q4s -lm /tmp/tmpwaazorcm.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmpkv6f6q4s'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:27:00.260572
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : 2ed65e431bfbcab3168ae4f4590fc2a6ccc1448853abf8174ff0fced595e54c8
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmp50exhn5p -lm /tmp/tmprztzimz9.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmp50exhn5p'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:43:09.562829
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : 4000e36d81b7ad1886b0430b9bccdc4f973f7d488b98f11a1528e1d2f8009167
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmp3k608a5f -lm /tmp/tmp7wm2qne0.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmp3k608a5f'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:48:33.328930
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : 80bee8cfe4305db665a79590bafc927ad89865710fad0e93acbc59a26de0f525
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=[])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=[])
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:49:27.375315
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : 1370b8b30920376a8de2c875365a7c79ec2168c1e968882b4dbe18d8f5fa6064
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmp5tum_jh1 -lm /tmp/tmpxlk4p3nz.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmp5tum_jh1'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:56:16.392146
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : b68d0d3b050bc18c09ed12970cec9dcf9d9b8c3816b2c77b4a66cce73167bbd7
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun -N6', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun -N6', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
usage: e4s-cl init [-h] [--system machine] [--launcher launcher]
                   [--launcher_args launcher_args] [--mpi /path/to/mpi]
                   [--source script] [--image /path/to/image] [--backend technology]
                   [--profile profile_name] [--wi4mpi path] [--wi4mpi_options opts]
e4s-cl init: error: argument --launcher: Launcher argument 'srun -N6' could not be resolved to a binary
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:59:12.937213
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : f8be31d20e3ad8fe2ea5d3b133dbc6f021d2492d714ae70f3b07940e11a1dfd5
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '-N', '3', '-n', '3', './a.out'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '-N', '3', '-n', '3', './a.out'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '-N', '3', '-n', '3', './a.out'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '-N', '3', '-n', '3', './a.out'])
Failed to determine necessary libraries: program exited with code 156
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:16:57.733090
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : 862ab45a8cf14653ac482ca977a491d90bde64fae9b391f1ad1cb12c9908efb4
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '3', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '3', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '3', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '3', './c_hello.x'])
Received keyboard interrupt. Exiting.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:17:22.943267
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : faa959952f2b5fbb27253e987303baba50cf930b7a424fb5e1bd79974c39def9
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '164', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '164', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '164', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '164', './c_hello.x'])
Failed to determine necessary libraries: program exited with code 1
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:17:39.959255
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : d6737d8a717d7421c5ca98fb8ab422465f16c489e0b2e351b881ee905c8383b2
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-n', '164', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-n', '164', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-n', '164', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-n', '164', './c_hello.x'])
Received keyboard interrupt. Exiting.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:18:26.050153
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : 4e6b9d8703ee647fa200ec47dda7a9dd807aee6690e8e05a5545d8aed37b6413
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'])
Failed to determine necessary libraries: program exited with code 244
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:23:17.684747
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : 063a179af243a7467b11d2e2a95c37ecdd23a59cbb46420f8fd007f33addc0d9
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'])
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:28:35.830914
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : 50f37f205e994b6ca01871c9c7064573903390a37ec908596a7e2c811fffe009
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'])
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:30:16.899563
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : d274bdec7f5d2ed4e9c1150e2ddd1f7930fdcf021231c23c2631ce57a563c353
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'])
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:30:52.826689
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : dae948381ffebe02644147d6fbd13a77c094c577909aa8e5038c6e917eed7ccc
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity', '--profile', 'pw'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity', '--profile', 'pw'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity', profile_name='pw')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmpgwhuuqtz -lm /tmp/tmphp09k8es.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmpgwhuuqtz'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug e4s_cl.error:168] Traceback (most recent call last):
  File "/contrib/bin/e4s-cl", line 12, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/__main__.py", line 200, in main
    return cli.execute_command([cmd], cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/__init__.py", line 314, in execute_command
    return find_command(cmd).main(cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/init.py", line 436, in main
    controller.update({'name': requested_name}, profile.eid)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/model/profile.py", line 102, in update
    super().update(data, keys)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/mvc/controller.py", line 236, in update
    database.update(data, keys, table_name=self.model.name)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cf/storage/local_file.py", line 459, in update
    table.update(fields, doc_ids=[keys])
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 340, in update
    self._update_table(updater)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 709, in _update_table
    updater(table)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 337, in updater
    perform_update(table, doc_id)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 326, in perform_update
    table[doc_id].update(fields)
KeyError: 2

An unexpected KeyError exception was raised:

2

Traceback (most recent call last):
  File "/contrib/bin/e4s-cl", line 12, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/__main__.py", line 200, in main
    return cli.execute_command([cmd], cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/__init__.py", line 314, in execute_command
    return find_command(cmd).main(cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/init.py", line 436, in main
    controller.update({'name': requested_name}, profile.eid)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/model/profile.py", line 102, in update
    super().update(data, keys)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/mvc/controller.py", line 236, in update
    database.update(data, keys, table_name=self.model.name)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cf/storage/local_file.py", line 459, in update
    table.update(fields, doc_ids=[keys])
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 340, in update
    self._update_table(updater)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 709, in _update_table
    updater(table)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 337, in updater
    perform_update(table, doc_id)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 326, in perform_update
    table[doc_id].update(fields)
KeyError: 2

This is a bug in E4S Container Launcher.
Please raise an issue on Github with the contents of '/home/Thomas.Robinson/.local/e4s_cl/logs/debug_log'.

Let me know if there's any other information you need

Profile name handling doesn't escape regex special characters

When using a profile whose name contains regex special characters, the application doesn't recognize it from the list of known profiles:

$ e4s-cl profile list
== Profile Configurations (/storage/users/fdeny/.local/e4s_cl/user.json) ====================================================

+----------+-------------------------+---------+-------+-----------+-------+--------+
| Selected |          Name           | Backend | Image | Libraries | Files | WI4MPI |
+==========+=========================+=========+=======+===========+=======+========+
|    *     | Intel(R)MPI_2019Update6 |     N/A |   N/A |    21     |  16   |   No   |
+----------+-------------------------+---------+-------+-----------+-------+--------+

$ e4s-cl profile edit --backend singularity
usage: e4s-cl profile edit <profile_name> [arguments]
e4s-cl profile edit: error: argument profile_name: Pattern 'Intel(R)MPI_2019Update6' does not identify a single profile: 0
profiles match

Library support dropped by source

Some libraries hang when used with the new sourcing launch method. A fd or environment variable may be erased or not passed along.

Libraries affected:
intelmpi - 2021.1-beta09

Improve file referencing in `profile edit`

/path/ and /path are considered as different. If they are removed but don't exist, no error message is returned.

Launching scripts fails - `invalid ELF header`

In case of incompatibility between glib versions, e4s-cl will try and overload the linker in the container. The resulting script has the following format:

[...]

# Finally run the command, using the imported linker if applicable
%(linker)s %(command)s

In case %(linker)s is empty, nothing is wrong. But in the cases it is not, %(command)s must be a binary, or the following appears (#28):

PID on HOST: COMMAND: error while loading shared libraries: COMMAND: invalid ELF header

Steps taken to endure the correct linker is used also include its addition to LD_PRELOAD.

'--' is not parsed when launcher is implemented

Errors parsing launcher CLI

WI4MPI's launcher expects underlying launcher arguments to be passed to the -E option in a string. To use OpenMPI's --oversubscribe, one should use:

mpirun -E '--oversubscribe' -np 8 -from mpich -to openmpi ./bin/lu.A.8.mvapich2

This string can contain spaces, which may confuse the current parsing system. Using shlex or a similar dedicated parser might resolve this issue.

Ubuntu ppc64le `libc` not resolved

The linker implementation does not take into account the rules in /etc/ld.so.conf and fails to resolve the libc file on Ubuntu ppc64le.

Creation of default system-tailored profile

e4s-cl should be able to create a default profile for the system MPI, along with a backend and image, for ease of use.

Format error in builtin system crashes

e4s-cl init --system fails when the builtin profile has a format error. A format check when importing a profile (in packages/e4s_cl/cf/assets.py:add_builtin_profile) should be enough to solve this.

Sort libaries by their opening methods

Using profile detect, some libraries are categorized as files, as their are opened by their path without using the dynamic linker and are not in the library cache. However, some files are in the standard search paths, like on cyclops libmlx5, and must be categorized as libaries.

Source files in container not supported

When a profile is given a script to source from inside the container, e4s-cl raises an error as the file is not found on the system. This is an artifact from the potential script optimization idea that was thrown around.

Allowing for files not found on the system should be supported, even if a warning is raised to warn the user if it is a mistake.