equinor / komodoenv

Komodoenv is the virtualenv of the Komodo world

Komodoenv

Komodoenv is to a Komodo release as what virtualenv is to a Python installation. Komodoenv creates a user-modifiable komodo environment based on an existing komodo release, allowing users to install additional or updated packages without the necessity for ugly hacks.

Komodoenv uses virtualenv at its core.

Usage

Create

The easiest way to create a komodoenv is to first enable a komodo release.

$ source /prog/res/komodo/stable/enable
$ komodoenv my-kenv

Now you are able to source my-kenv:

$ source my-kenv/enable
$ which ert  # => [..]/my-kenv/root/shims/ert

$ # Install other packages, which you can use directly, or eg. in an ERT forward model
$ pip install my-package

Note that the newly created my-kenv is a fully-fledged komodo release, meaning you don't need to enable the original before enabling my-kenv. In fact, enabling my-kenv will disable the other komodo release.

Update

Komodoenv doesn't automatically update your environment. It does check if there's an update when enabling, and you'll often be able to run the komodoenv-update command to update your environment to use the latest komodo release packages.

Development

Installing

Komodoenv is meant to be part of a komodo release. As such, it is not meant to be installed by users directly.

This project requires Python 3.8. Then, install this project with pip install .

Testing

Komodoenv uses pytest for test running. Ensure that it's installed with pip install pytest. To run tests:

pytest tests/

komodoenv's Issues

Julevask: Migrate to Python3.8

According to https://github.com/equinor/scout/issues/687 we should migrate to Python3.8 at least.

Add MOTD

Part of https://github.com/equinor/scout/issues/246

Also see equinor/ert#735 and equinor/komodo#131 and equinor/komodo#140

FileNotFoundError in komodoenv-update

Ran

komodoenv -r stable foo

Get this (seemingly randomly)

Traceback (most recent call last):
  File "/private/jond/foo/root/bin/komodoenv-update", line 436, in <module>
    main()
  File "/private/jond/foo/root/bin/komodoenv-update", line 403, in main
    if args.check and not can_update(config):
  File "/private/jond/foo/root/bin/komodoenv-update", line 247, in can_update
    config, os.path.join(config["komodo-root"], config["tracked-release"], "root")
  File "/private/jond/foo/root/bin/komodoenv-update", line 232, in get_pkg_version
    for name in os.listdir(pkgdir):
FileNotFoundError: [Errno 2] No such file or directory: '/prog/res/komodo/stable-py36/root/lib/python3.6/site-packages'

Affects bleeding. I have not seen the same after sourcing the equivalent 2021.02.00-py36-rhel7.

Long-lived komodoenvs

One of the ambitions of the komodoenv project was to have a long-lived komodoenv that would "track" a komodo release, usually stable, and update accordingly.

Does this work in practice? Are people using this?

This issue can be closed if:

1. We have someone who uses this functionality and it works well.
2. OR we find out people are not using this because it's buggy. If so, create an issue fixing that and recreate this issue after some time.
3. OR we find out people are not using this functionality at all, in which case it can be considered "working well".

Don't crash when Python doesn't have ctypes installed

We use ctypes to test whether the user is on NFS or not via a Linux-only function: https://github.com/equinor/komodoenv/blob/master/komodoenv/statfs.py

This is just a nice feature for the user, and we shouldn't crash if the user is missing the ctypes module (which can happen on particularly thin Python installs)

System version warning when using rhel8

Describe the bug
In 'update.py' the distro version is currently hardcoded to be 7.0.0 which will create issues when moving to rhel8. Also it seems like distro is not installed on any of the rhel-machines, so the part of the code which uses that with the default system python3 should probably be rewritten.

To Reproduce
Create komodoenv on RHEL8.

Expected behavior
Should not produce warning for RHEL8.

Run tests also with python3.11

The actual tests explicitly tests for python3.8 by mocking a komodo. Now that we are building releases for python3.11 we should also test with python3.11.

Update pythonX_path fixtures for ubuntu 20.04

python 3.6 is no longer shipped with ubuntu-latest (20.04) but comes with python 3.8. The github workflow should be adjusted accordingly. In addition the fixture for getting the python executable is no longer working, as python 3.6 is no longer installed in /usr/bin

Should "out-of-date" komodoenv result in exit code 1?

Given that your komodoenv becomes outdated, the sourcing will result in a sys.exit with a message letting you know you should update.

komodoenv/komodoenv/update.py

Line 412 in b35f782

sys.exit(

Should this be an exit code of 1?

Use case where this can be challenging: running this as part of ERT will make the forward model job fail even before status's are reported back.

`komodoenv` assumes `subprocess` `/bin/bash` only prints output from command

I have one line printed when a new terminal is started, resulting in the assert statement on line 71

komodoenv/komodoenv/__main__.py

Lines 59 to 71 in 45d0ccd

	lines = (
	subprocess.check_output(
	[
	"/bin/bash",
	"-c",
	f"source {root / name / 'enable'};which python;python --version",
	]
	)
	.decode("utf-8")
	.splitlines(keepends=False)
	)
	assert len(lines) == 2

failing (3 !== 2).

Could one easy fix be to relax it to >= 2 and then use the last two output lines (i.e. effectively ignoring any start-up messages the user has in bash)?

virtualenv from bleeding-py36 may conflict with ~/.local/python3.6/site-packages/virtualenv

Steps to reproduce:

1. pip install --user --upgrade 'virtualenv<20'
2. source /prog/res/komodo/bleeding-py36/enable
3. komodoenv -rstable-py36 kenv

Observe error:

KomodoType.SHIM [3, 6, 9, 'final', 0]
Traceback (most recent call last):
  File "/prog/res/komodo/bleeding-py36/root/libexec/komodoenv", line 11, in <module>
    sys.exit(main())
  File "/prog/res/komodo/bleeding-py36/root/lib/python3.6/site-packages/komodoenv/main.py", line 59, in main
    create(ctx)
  File "/prog/res/komodo/bleeding-py36/root/lib/python3.6/site-packages/komodoenv/creator.py", line 168, in create
    create_virtualenv(ctx)
  File "/prog/res/komodo/bleeding-py36/root/lib/python3.6/site-packages/komodoenv/creator.py", line 101, in create_virtualenv
    str(ctx.dstpath / "root"),
  File "/private/zom/.local/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 24, in cli_run
    session = session_via_cli(args, options)
  File "/private/zom/.local/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 32, in session_via_cli
    parser, elements = build_parser(args, options)
  File "/private/zom/.local/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 60, in build_parser
    SeederSelector(interpreter, parser),
  File "/private/zom/.local/lib/python3.6/site-packages/virtualenv/run/plugin/seeders.py", line 8, in __init__
    possible = self.options("virtualenv.seed")
  File "/private/zom/.local/lib/python3.6/site-packages/virtualenv/run/plugin/base.py", line 39, in options
    cls._OPTIONS = cls.entry_points_for(key)
  File "/private/zom/.local/lib/python3.6/site-packages/virtualenv/run/plugin/base.py", line 18, in entry_points_for
    return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
  File "/private/zom/.local/lib/python3.6/site-packages/virtualenv/run/plugin/base.py", line 18, in <genexpr>
    return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
  File "/private/zom/.local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 105, in load
    module = import_module(match.group('module'))
  File "/prog/res/komodo/bleeding-py36/root/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

Notice how it's finding my local virtualenv. If I run with export PYTHONNOUSERSITE=1 then a komodoenv is created as expected.

Remove messages about not being able to uninstall packages from `/prog/res/komodo`

Pip install does:

1. Install package into the user's komodoenv
2. Try to remove the old package

Well, if the package is installed into /prog/res/komodo proper then it won't be able to remove the package due to correctly missing permissions.

It's weird that it tries at all. This issue should be considered resolved if any of these are done:

1. We patch pip so that it doesn't try to do anything with /prog/res/komodo at all
2. We patch pip so that it doesn't output warnings about not having permissions

not providing -r yields misleading error message

(bleeding-py36) eism@st-linrgsn183:~> komodoenv test
ERROR:root:Could not automatically detect active Komodo release. Either enable a Komodo release that supports komodoenv or specify release manually with the `--release' argument.
usage: komodoenv [-h] [-r RELEASE] dest
positional arguments:
  dest
optional arguments:
  -h, --help            show this help message and exit
  -r RELEASE, --release RELEASE
Traceback (most recent call last):
  File "/prog/res/komodo/bleeding-py36/root/bin/../libexec/komodoenv", line 11, in <module>
    sys.exit(main())
  File "/prog/res/komodo/bleeding-py36/root/lib/python3.6/site-packages/komodoenv/main.py", line 53, in main
    root = (KOMODO_ROOT / args.release).resolve()
  File "/prog/res/komodo/bleeding-py36/root/lib/python3.6/pathlib.py", line 899, in __truediv__
    return self._make_child((key,))
  File "/prog/res/komodo/bleeding-py36/root/lib/python3.6/pathlib.py", line 686, in _make_child
    drv, root, parts = self._parse_args(args)
  File "/prog/res/komodo/bleeding-py36/root/lib/python3.6/pathlib.py", line 640, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType```

Solution could be to improve the error message and assume -r from what is enabled.

Editable installs of packages installing into komodo proper don't work

If one pip installs with --editable, doing pywhich X will point to the old version of the package rather than the new.

Renaming the file to something that comes late alphabetically will mean it gets found last, and thus any overridden package is found earlier.

Add up-to-date check

The komodo-update script should allow for --check flag when the user enables the environment, which tells them to upgrade, ie:

Your custom komodo environment is out of date. Current
`stable` is version 2020.08.02, while your environment
uses 2020.07.01. To upgrade, run:

$ komodo-update

Remove the need for Python.h workaround when using RH Py 3.8

Bottom of https://fmu-docs.equinor.com/docs/komodo/equinor_komodo_usage.html

NB if you are extending a Python 3.8 environment: If you are using a python package the builds from source and requires the Python development headers (Python.h) , then the include folders needs to be passed as build options to pip manually. The following command shows how:
pip install --global-option=build_ext --global-option="-I/opt/rh/rh-python38/root/usr/include/python3.8" <your_package>

Komodoenv prevents import of google.protobuf

Repro:
produce komodoenv based off of a recent release (> May 2020)
python -c "import google.protobuf"

What happens:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named google.protobuf

What should happen:
It should import

Provide a better message when in a non-symlinked komodo release

komodoenv will not work if you are in a non-symlinked komodo release. That is, if you've sourced stable-py36, then simply doing komodoenv kenv will create it based on whatever specific version stable-py36 is, and mark the environment as targetting stable-py36.

However, suppose you source 2020.10.b8-py36 and none of stable, testing, unstable or bleeding point to it. I considered this to be dangerous, so I decided not to do anything in this case, and have komodoenv fail unless you specify a release with --release=.

1: Is this a good idea?
2: If it is, provide a message that will tell the user of the dangers and suggest running komodoenv, but with --release specified.

---

komodoenv/komodoenv/__main__.py

Lines 60 to 62 in 2ebcd9d

	suffix = match[1]
	for mode in "stable", "testing", "unstable", "bleeding":
	dir_ = root / (mode + suffix)

Not auto-detecting komodo release version

(bleeding-py36) [zom@st-linapp1192 ~]$ komodoenv kenv
/prog/res/komodo/stable-py36
/prog/res/komodo/testing-py36
/prog/res/komodo/unstable-py36
/prog/res/komodo/bleeding-py36
Traceback (most recent call last):
  File "/prog/res/komodo/bleeding-py36-rhel7/root/bin/komodoenv", line 8, in <module>
    sys.exit(main())
  File "/prog/res/komodo/bleeding-py36-rhel7/root/lib/python3.6/site-packages/komodoenv/__main__.py", line 157, in main
    args = parse_args(args)
  File "/prog/res/komodo/bleeding-py36-rhel7/root/lib/python3.6/site-packages/komodoenv/__main__.py", line 126, in parse_args
    args.release = detect_rhel(args.release)
  File "/prog/res/komodo/bleeding-py36-rhel7/root/lib/python3.6/site-packages/komodoenv/__main__.py", line 75, in detect_rhel
    if not komodo_prefix.is_dir():
AttributeError: 'NoneType' object has no attribute 'is_dir'

Detect the komodo root from PATH

If Komodo releases are not located in /prog/res/komodo then trying to automatically use an env for a new komodoenv results in an error like:

/prog/res/komodo/2023.02.b3-py38-rhel7' is not a valid komodo release

In my case, the root is /prog/komodo/2023.02.b3-py38-rhel7.

Perhaps it's best to keep things super explicit, but it might be nice to try detecting the komodo root from the combination of KOMODO_RELEASE and PATH.

For reference, here's how we do this in komodo. (It would be even nicer if Komodo set an environment variable with the path to the root -- see linked issue.)

Komodoenv does not find the correct python when using rhel8 and python 3.11

Describe the bug
When creating a komodoenv with a komodo built for rhel8 and python3.11, the associated python defaults to the system python3.6.

Additional context
A workaround here seems to be to explicitly specify python3.11 instead of writing merely python (see below), but maybe a better solution can be found.
subprocess.check_output( [ str(self.srcpy.executable)+str(self.srcpy.version_info[0])+"." + str(self.srcpy.version_info[1]), "-m", "venv", "--copies", str(self.dstpath / "root"), ], env=env, )

Pytest fails when filterwarnings are present in configuration

Tested with komodo version 2020.08.00-py36, steps to reproduce:

Create a minimal test, test_minimal.py:

import pandas

def test_something_simple():
    pass

Create pytest.ini:

[pytest]
filterwarnings =
    error

Fails with:

RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

Add csh support

Komodoenv only creates a bash enable script. A csh enable script should be created.

Make some elements of the CLI clearer

During build

We should change the color of the notice about komodoenv being in beta. I think red suggests error, but this is really just information. I think it should be blue, or possibly yellow.

It could also be good to know what "issues" are, how often "once in a while" is, and to change the "this'll" to "this will" and "singular" to "single" (singular usually means 'good' or 'unique').

After build

The build process finishes with:

Komodoenv has successfully been generated. You can now pip-install software.

    $ source /tmp/mtha/kenv/enable

I think it should emphasize that after activating the environment you can pip-install software.

Notify the user about the alpha state of the program

When you run komodoenv, it should print out some kind of disclaimer about being early in development.

Uninstalling/masking packages from komodo

There should be a way to mask packages installed into komodo so that they're not usable in komodoenv. Perhaps by having a list of packages that are "removed" and have a custom site.py remove them.

I am very much unsure whether this can be done in a nice way. Adding it to the stable milestone, but this issue can be dropped easily.