llnl / uberenv Goto Github PK

If uberenv is set to clone a specific commit of spack, moving to a branch will cause the new ref to be fetch but not checked out.

So removing spack_commit: ... and optionally adding spack_branch: ... in an instance with spack already cloned will have no effect.

That’s because the implementation will only use the branch in the clone operation, which is a fetch if the clone already exists.

There is a repo_pull option to force update of a spack branch each time uberenv is called. This option is not documented. It seems weird to have to setup an option fetch a branch.

Here is what I suggest:

• OK, using a branch to point to spack is unstable, and not desirable in production.
• but pointing to a branch should imply keeping it up-to-date, at least by default, this would allow to switch from one branch to another, and fix this issue.
• repo_pull should therefore be changed to do_not_pull, so that it is explicit in the configuration that we enforce one particular ref

as part of normal output.

Several uberenv users have had issues when mirroring against branches where the current version of a branch will get cached.

Spack has an option to skip unstable versions when creating mirrors:

> spack mirror create --create-unstable-versions

Uberenv should use this by default, and possibly expose this as a user-facing option

it makes life easier to have an option to store it here, vs always pass on the command line

the --spack-build-mode option is None here by default:
https://github.com/LLNL/uberenv/blob/main/uberenv.py#L168

but later on, it's set to its default value (dev-build):
https://github.com/LLNL/uberenv/blob/main/uberenv.py#L573-L575

i tried setting the default to dev-build in the parser.add_option, since it seemed more intuitive, but it always made the build mode dev-build, even when i set the build mode flag in uberenv's config file to install.

users aren't affected so not sure if this is something worth investigating but i figured i'd point it out

we want one

We currently only have CI tests for the uberenv-pkg build mode. We should be testing dev-build and install as well.

documenting while still in my brain cache:

• we have the proper dance for using envs in a script, we should use {path/to/spack/bin/spack} -e .
• we will need to use env views to achieve the equivalent of spack activate for python packages
  • does this have implications on the generated host configs?
• externals need explicit versions with clingo (old configs will need to be updated)

self.spec_hash seems to only be set in show_info()

which I think implies that using it later to locate an install will fail, like in find_spack_pkg_path_from_hash unless you call show_info again after the install (or call uberenv again?)

find_spack_pkg_path using name and spec might be more robust

uberenv calls spack with a string that looks like:

{package_name}{spec}

when using --spec arg, folks often forget the "%" which leads to confusing error message.

We should check if the passed spec doesn't start with "%" and warn or make things right.

The format should follow Keep a Changelog

See Axom for an example: https://github.com/LLNL/axom/blob/develop/RELEASE-NOTES.md

I noticed that we are adding the spec to the Spack environment and then also adding it to the various Spack install commands here:

https://github.com/LLNL/uberenv/blob/main/uberenv.py#L987

I am pretty sure this does not need to happen. This should be enough, but I haven't tested it in all the build modes:

spack add foo
spack install

I've noticed this error in CI: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

May be a useful thing to add (see https://spack.readthedocs.io/en/latest/command_index.html#spack-concretize)

I figured I leave this as an issue to reduce the scope of the spack environments PR (#96).

TODO:

• see if spack environment concretizes again after having a different spack.yaml with/ without --force
• add as command line option

@cyrush @kennyweiss

I need clarifications about the expected behavior here:

• Apparently the goal is to provide a list of packages (dependencies) that need to be uninstalled before installing again. right?
• Is it supposed to work if 2 variations of the main package are present?

Example:
Let’s say I ran

> python script/uberenv/uberenv.py
> python script/uberenv/uberenv.py --spec=+debug

I would imagine the goal would be that if I run again

> python script/uberenv/uberenv.py

Only the non-debug dependencies would be removed and then reinstalled, right?

This needs to be changed to support the new way of handling the same functionality.

https://github.com/spack/spack/blob/4a8db0069186f2bb47198ef868131b6ea075323f/lib/spack/spack/cmd/activate.py#L30-L31

per @adrienbernede

spack installs clingo in ~/.spack, and does that with spack-like install tree, coming with a database.
That would be OK if clingo was installed once for all, and the database version never changes.
There was a recent change in Spack, and all databases moved from yaml to json.
​
This means, between two projects using a different version of spack, you may have a nice error because of a conflict on your database version in ~/.spack.
​
Work around 1: (and typically a new feature for Uberenv): always set the store location (bootstrap clingo install) to a custom location. That may be a bit painful because installing clingo is painful.

Work around 2: Have LC install a clingo we can use. (We may still need work around 1, just to be safe)

I found myself in the situation where uberenv would appear to ignore concretizer: original in the configuration file.

It turned out that since I point to spack@develop support for the original concretizer was already dropped.

I am guessing Uberenv could check spack version to warn about that.

Proposed workflow:

uberenv.py --spack-create-environment /some/path/spack.yaml

Which calls the following commands:

// setup spack
spack env create -d /some/path/spack.yaml
spack compiler find
spack external find --not-buildable <list of important packages coming from .uberenv_config.json>

It would be helpful if Uberenv's docs had a brief section about how users can use the copy of spack (or vcpkg) that uberenv has configured.

Once uberenv is using spack environments (#18) this should be added to the docs.

I'm using --upstream to point to a spack installation and I'm not getting the error that an upstream wasn't found but then the upstream isn't used and the dependencies are built instead. Is there something else I need to do to get it to work?

I'm passing --upstream=/path-to/spack/opt/spack to the script and upstreams.yaml is created in uberenv_libs/spack/etc/spack/defaults/ with the correct path to the upstream spack installation. Any ideas what I'm doing wrong?

As part of debugging some compiler flag settings, I'd like a command-line option to increase output verbosity to see what flags are being passed to the build systems spack is invoking during spack dev-build. If I were to file a PR adding a --verbose flag to uberenv, would that code be of interest?

Is there any roadmap for logging features in uberenv? I imagine there might be other phases of provisioning worth logging, but haven't yet reviewed the codebase with that thought in mind.

If you've installed multiple versions of say py-scipy you need to specify which one you want to activate, it would be nice if Uberenv could pick out the exact version that was used, or at least use the same compiler to pick out the right package.

==> lvarray: Executing phase: 'hostconfig'
[activating dependent packages]
[exe: spack/bin/spack activate py-scipy]
==> Error: py-scipy matches multiple packages.
  Matching packages:
    km2gnzw [email protected]%[email protected] arch=linux-rhel7-x86_64
    db5qgqb [email protected]%[email protected] arch=linux-rhel7-x86_64
  Use a more specific spec.

Currently, uberenv copies the spack configuration files for the appropriate platform from the user's source repo. This works well, so long as the needed tools are in standard locations. When this is not the case, users can use the --upstream option to chain spack installations (https://spack.readthedocs.io/en/latest/chain.html). This allows the user to specify extra packages that have been built by an external version of spack. However, there is no way to add extra packages that are in custom locations on a user's machine. For example, on my mac, the pre-installed cmake and doxygen are older than what I need. Rather than having spack build these, I downloaded them and put in /Users//Applications/. It would be nice if I can tell spack about these while still using a project-specific scripts/spack/configs/darwin/.

Proposal: add a --config-scope option to leverage spack's equivalent feature (https://spack.readthedocs.io/en/latest/configuration.html#custom-scopes) just like --upstream is currently supported.

With the above, a user could easily have a separate configuration with tools that were installed manually rather than through spack.

As an additional option, uberenv could also support --use-user-config to use the configuration found at ~/.spack. This would be equivalent to --config-scope ~/.spack/<platform>. This would be convenient for users who user their user-specific configuration files to just keep track of custom tools that are not affected by the compiler (e.g. perl, python, cmake, doxygen, flex, bison, etc).

For example, spack config next to uberenv.py, config file, then command line option.

When things go wrong in spack land, the first request from spack folks is the stack trace obtained with:

spack --debug --stacktrace <command>

We should have a spack_debug option that allows users to easily find this info vs hacking it in.

Optparse is deprecated.

A lot of this could be moved into functions between the various places it's done (git cloning for one)

In SpackEnv's patch function, we manually copy all of the packages in to spack's builtin package repo, but it would be better to get the location of the project's package repo (in something like --spack-package-repo-dir), and do spack repo add <spack_package_repo_dir> after creating the spack environment.

This would also involve creating a repo.yaml for the ci test project.

I was thinking about doing this after the environment's PR #96

https://spack.readthedocs.io/en/latest/repositories.html

right now, if you aren't running on LC (SYS_TYPE env set) or macos (darwin) you will get the error message:

[ERROR: Given path for 'spack_configs_path' does not contain platform directories: ...]

The logic here is wrong:

if platform is None, logic should simply skip trying to find the defaults for that platform.

(I hit this in a docker container)

I would like the ability to build an arbitrary package in uberenv that is not the project package. I think this would work with the current system if we just detect if the first character given in the spec command line option is either a letter or not one of the special characters reserved by Spack specs (@^%+~).

For example:

uberenv.py --spec="[email protected]"

This would be useful for not only needing a random Spack package installed but also debugging subsets of your build when things go wrong.

Uberenv is being used in more and more projects, especially for CI purpose.

However, the fact that is has to be added in-source is a brake to broader adoption and ease of use.

Adapting uberenv to be usable as a submodule will probably:

• break the current workflow
• require multiple small changes

Still, I think this should be done. If we do it in a separate branch at first, we may have trouble migrating to Spack environments afterward (duplicate the amount of work).

So I’m considering completing my work on environments first, even if some changes in Spack will likely imply adapting uberenv later on, hopefully drastically reducing its scope.

Input welcome

need to change:

https://github.com/LLNL/uberenv/blob/master/uberenv.py#L608

to include the full spec, not the pkg name

When spack_commit is present in the project.json file, uberenv uses that commit of spack.

It currently uses git reset --hard $commit, which resets the branch pointer for spack's develop branch to the desired commit. Unfortunately, this makes it look like it is still using the develop branch, which could be confusing to users.

Instead, it should use git checkout $commit, which puts spack in a detached head state indicating the current commit.

Hey,
Is there a way to make uberenv use an existing spack installation without fetching? I'd like to use uberenv in situations where I'm unable to fetch from github.
Thanks!
Jun

We need to move away from the old Spack usage of a split packages/compilers file (I don't think there is a word for it) and to Spack Environments.

I had a long conversation with @becker33 and this was his recommendation to convert:

# Translation of dev-build -u X in uberenv-speak
spack env create env /path/to/environment/file
spack -e env add $spec
# dev-build mode command
spack -e env develop -u final_phase--no-clone --path $location_of_axom $spec  # remove -u final_phase for install
# install and uberenv-pkg modes command
spack -e env install -u final_phase --no-add $spec  # remove -u final_phase for full install

Note this example is for dev-build with a final phase mode but has alterations for install and dev-build with no final phase.

Spack has started to find packages in other directories when --reuse is not on. We need to add --fresh due to a default being flipped to stop this from happening.

This commit spack/spack@ae6e83b#diff-4659603983779f9e0eb6449daeb01d7077214ec46e6285a3e9be51e419372baf on October 26, 2021 breaks uberenv's modification of spack/lib/spack/spack/config.py. The develop branch of spack is no longer compatible with uberenv.

The uberenv function at fault

    def disable_spack_config_scopes(self,spack_dir):
        # disables all config scopes except "defaults", which we will
        # force our settings into
        spack_lib_config = pjoin(spack_dir,"lib","spack","spack","config.py")
        print("[disabling config scope (except defaults) in: {0}]".format(spack_lib_config))
        cfg_script = open(spack_lib_config).read()
        for cfg_scope_stmt in ["('system', os.path.join(spack.paths.system_etc_path, 'spack')),",
                            "('site', os.path.join(spack.paths.etc_path, 'spack')),",
                            "('user', spack.paths.user_config_path)"]:
            cfg_script = cfg_script.replace(cfg_scope_stmt,
                                            "#DISABLED BY UBERENV: " + cfg_scope_stmt)
        open(spack_lib_config,"w").write(cfg_script)

Don't have a good long term solution right now that preserves the original intentions, but I do have a workaround.

diff --git a/uberenv.py b/uberenv.py
index 2570c8a..162d1a9 100755
--- a/uberenv.py
+++ b/uberenv.py
@@ -719,9 +719,9 @@ class SpackEnv(UberEnv):
         spack_lib_config = pjoin(spack_dir,"lib","spack","spack","config.py")
         print("[disabling config scope (except defaults) in: {0}]".format(spack_lib_config))
         cfg_script = open(spack_lib_config).read()
-        for cfg_scope_stmt in ["('system', os.path.join(spack.paths.system_etc_path, 'spack')),",
-                            "('site', os.path.join(spack.paths.etc_path, 'spack')),",
-                            "('user', spack.paths.user_config_path)"]:
+        for cfg_scope_stmt in ["    configuration_paths.append(\n        ('site', os.path.join(spack.paths.etc_path, 'spack')),\n    )", "    if not disable_local_config:\n        configuration_paths.append(\n            ('user', spack.paths.user_config_path)\n        )"]:
+            cfg_script = cfg_script.replace(cfg_scope_stmt, "#DISABLED BY UBERENV: ")
+        for cfg_scope_stmt in ["('system', os.path.join(spack.paths.system_etc_path, 'spack')),"]:
             cfg_script = cfg_script.replace(cfg_scope_stmt,
                                             "#DISABLED BY UBERENV: " + cfg_scope_stmt)
         open(spack_lib_config,"w").write(cfg_script)

The used to be placed in the prefix dir, but now they placed in the dir where uberenv is run?
(I think we may have discussed this before)

I often use a few uberenv dirs -- I don't want the generated host configs to write over each other.

Especially with the new strategy to stop at the hostconfig stage, b/c the install dir doesn't even exist -- if the host-config is written over, its gone.

An option would be fine if folks are relying on this new behavior.

Had to change this in serac so figured we should update Uberenv as well with the new versions.

From a serac github action warning:
Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: actions/checkout@v2, docker/setup-buildx-action@v1, docker/login-action@v1, docker/build-push-action@v2, actions/upload-artifact@v2. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/

Uberenv's read-the-docs has not updated in almost 2 years.
See: https://uberenv.readthedocs.io

We should look into this and ensure that a new docs build is triggered on changes to uberenv.

We should also add a docs badge to uberenv's README.md

Things I plan to improve:

• Copying config.yaml as limits: new parameters have been added to spack config that are not set in old config.yaml files. -> Update config files or use commands to change config
• If I have a local fork of a spack package and I remove it, it won’t be deleted from spack.
• Mechanism to retrieve the host-config file produced by package could be improved.

Things to investigate:

• Apparently even with uberenv spack is not completely isolated from user config, (providers?).

you can pass a yaml file with an environment spec directly to spack.

spack install ./spack.yaml

This path may open up a much easier way to use envs than spack shell support.

          We should add a documentation section with some guidance on what to do to convert from the old split file way to the new environment file. It isn't a lot of work but may be confusing.

Originally posted by @white238 in #96 (comment)