Conda pack does not work with Python 3.10 or 3.11 about conda-pack

[jtilly]

I can reproduce your issue with conda-pack 0.6.0 when pulling it from anaconda (but I don't think there's anything we can do about that).

I can't reproduce your issue with conda-pack 0.7.0 from conda-forge.

Can you please run and report the output from:

conda create -y -f -n python310forge -c conda-forge python=3.10 conda-pack
conda activate python310forge
conda info
conda list
which conda-pack  # just to make sure we're using the correct conda-pack executable
conda-pack --version
conda-pack

You can also run conda-pack with --ignore-missing-files, which will get rid of the error message (but might just kick the can down the road).

The python3.1 symlink isn't necessarily an issue. It comes with python3.10 by default.

Edit: I somehow missed your comment about the conda versions. Yes: I can reproduce your issue with conda 4.9.2 and conda-pack 0.7.0 from conda-forge. Adding the --ignore-missing-files flag fixes the issue for me and I can properly unpack the resulting environment without issues. So maybe try that if upgrading conda is not an option?

[lgorenstein]

Also observed on our HPC cluster using older versions of Anaconda
(2020.11, conda version 4.9.2, base python version 3.8.5)

[lgorenstein]

I do not think this is complete, though.

With --ignore-missing-files, the very same python310forge environment packed as expected:

$ conda-pack --ignore-missing-files 
Collecting packages...
Packing environment at '/home/lev/.conda/envs/cent7/2020.11-py38/python310forge' to 'python310forge.tar.gz'
[########################################] | 100% Completed |  8.4s

However, I have received an error message upon unpacking:

$ module load anaconda/2020.11-py38
$ mkdir $HOME/.conda/envs/cent7/2020.11-py38/XXX
$ tar -xf python310forge.tar.gz -C $HOME/.conda/envs/cent7/2020.11-py38/XXX
tar: lib/python3.1: Cannot create symlink to `python3.10': File exists
tar: Exiting with failure status due to previous errors

The resulting environment activated and seemed to work, but the untaring error looks a bit scary, and it would definitely spook people.

Looking into the tarball, it seems that there are a lot of lines with normal lib/python3.10/...., a line with lib/python3.1 -> python3.10, and then a bunch of

lib/python3.1/site-packages/conda_pack-0.7.0.dist-info/INSTALLER link to lib/python3.10/site-packages/conda_pack-0.7.0.dist-info/INSTALLER

(note the symlink to a file inside python3.1 in addition to the python3.1 itself). And this is causing the error message, because upon unpacking the lib/python3.1 gets created as a directory rather than as a symlink):

$ ls -ld ~/.conda/envs/cent7/2020.11-py38/*/lib/python3.1*
lrwxrwxrwx  1 lev itap  10 Dec 13 09:27 /home/lev/.conda/envs/cent7/2020.11-py38/python310forge/lib/python3.1 -> python3.10/
drwxr-sr-x 36 lev itap 210 Dec 13 09:27 /home/lev/.conda/envs/cent7/2020.11-py38/python310forge/lib/python3.10/

drwxr-sr-x  3 lev itap   3 Dec 13 10:18 /home/lev/.conda/envs/cent7/2020.11-py38/XXX/lib/python3.1/
drwxr-sr-x 36 lev itap 210 Dec 13 10:18 /home/lev/.conda/envs/cent7/2020.11-py38/XXX/lib/python3.10/

(note how the original environment has a symlink, but the extracted one has two distinct directories)

[jtilly]

I can't reproduce this issue. What happens when you run the following three lines?

conda create -y -n python310 python=3.10 conda-pack  # (you can also use mamba instead of conda here)
conda activate python310
conda-pack

This works on Linux and macos for me.

Based on the error message that you're getting, I suspect that you ran pip install <something> in your Python 3.10 environment before trying to run conda-pack.

For example, when I do

conda create -y -n python310pip python=3.10 conda-pack  # (you can also use mamba instead of conda here)
conda activate python310pip
pip install 'pip<22.3.1' 
conda-pack

I get

Collecting packages...
CondaPackError: 
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- pip 22.3.1:
    lib/python3.10/site-packages/pip-22.3.1-py3.11.egg-info/PKG-INFO
    lib/python3.10/site-packages/pip-22.3.1-py3.11.egg-info/SOURCES.txt
    lib/python3.10/site-packages/pip-22.3.1-py3.11.egg-info/dependency_links.txt
    + 3 others

This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).

which looks a lot like your error message.

You seem to be using conda-pack=0.6.0. Please use >=0.7.0, otherwise you'll run into #199.

[lgorenstein]

Hello Jan,

Thank you for looking at this! I did not run any pip installs, and I did exactly the three lines you said. Note: I do not have conda-forge in my channels, so using just the default channel I end up with conda-pack 0.6.0. which throwns an error:

$ cat ~/.condarc 
channels:
  - defaults

$ module --force purge
$ module load anaconda/2020.11-py38
$ conda create -y -n python310 python=3.10 conda-pack
[...]
conda-pack         pkgs/main/noarch::conda-pack-0.6.0-pyhd3eb1b0_0
[...]
$ conda activate python310
$ conda-pack
Collecting packages...
CondaPackError: 
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- wheel 0.37.1:
    lib/python3.1/site-packages/wheel-0.37.1-py3.9.egg-info/PKG-INFO
    lib/python3.1/site-packages/wheel-0.37.1-py3.9.egg-info/SOURCES.txt
    lib/python3.1/site-packages/wheel-0.37.1-py3.9.egg-info/dependency_links.txt
    + 20 others
- conda-pack 0.6.0:
    lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/INSTALLER
    lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/LICENSE.txt
    lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/METADATA
    + 19 others

This is usually due to `pip` uninstalling or clobbering conda managed files,
...

As you can see, the pip warning is actually a red herring (note the lib/python3.1/ instead of lib/python3.10/). So this probably looks like #199 you mentioned because default channel only has 0.6.0, not 0.7+. It would certainly be nice to have the most up-to-date conda-pack in the main conda channel, though.

---

But the problem is that I also see the same error with 0.7.0 from conda-forge:

$ conda create -y -n python310forge -c conda-forge python=3.10 conda-pack
[...]
  conda-pack         conda-forge/noarch::conda-pack-0.7.0-pyh6c4a22f_0
[...]

$ conda activate python310forge
$ conda-pack
Collecting packages...
CondaPackError: 
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- setuptools 65.5.1:
    lib/python3.1/site-packages/_distutils_hack/__init__.py
    lib/python3.1/site-packages/_distutils_hack/override.py
    lib/python3.1/site-packages/distutils-precedence.pth
...

Again, there hasn't been any pip-ing, so the pip message is mistaken. And again the error messages shows lib/python3.1 instead of 3.10.

Strangely, there is indeed this weird 3.1 symlink in the site packages:

$ ls -ld ~/.conda/envs/cent7/2020.11-py38/python310forge/lib/python3.1
lrwxrwxrwx  1 lev itap  10 Dec 13 00:39 python3.1 -> python3.10/
drwxr-sr-x 36 lev itap 210 Dec 13 00:39 python3.10/

I wonder if it's an artifact of a (somehwhat old) Anaconda? I did not get the error with conda-pack 0.7.0 when I used the latest downloaded Miniconda (with conda 4.12 as opposed to 4.9.2). OTOH, both conda versions create that 3.1 symlink, so somehow conda-pack is capable to trace thing in some circumstances?

[lgorenstein]

Thank you Jan. This being a centralized installation on an HPC cluster, updating its version of conda is, let's say, a process. So having a functional workaround with --ignore-missing-files for older Anacondas is extremely useful! I did run into problem during extracting the packaged tarball - please see my next post in this thread with detailed description.

---

For completeness sake, here's the report the commands you requested:

$ module load anaconda/2020.11-py38
$ conda create -y -f -n python310forge -c conda-forge python=3.10 conda-pack
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 22.11.1

Please update conda by running

    $ conda update -n base -c defaults conda


## Package Plan ##

  environment location: /home/lev/.conda/envs/cent7/2020.11-py38/python310forge

  added / updated specs:
    - conda-pack
    - python=3.10

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
  ca-certificates    conda-forge/linux-64::ca-certificates-2022.12.7-ha878542_0
  conda-pack         conda-forge/noarch::conda-pack-0.7.0-pyh6c4a22f_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.39-hcc3a1bd_1
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
  libgcc-ng          conda-forge/linux-64::libgcc-ng-12.2.0-h65d4601_19
  libgomp            conda-forge/linux-64::libgomp-12.2.0-h65d4601_19
  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
  libsqlite          conda-forge/linux-64::libsqlite-3.40.0-h753d276_0
  libuuid            conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000
  libzlib            conda-forge/linux-64::libzlib-1.2.13-h166bdaf_4
  ncurses            conda-forge/linux-64::ncurses-6.3-h27087fc_1
  openssl            conda-forge/linux-64::openssl-3.0.7-h0b41bf4_1
  pip                conda-forge/noarch::pip-22.3.1-pyhd8ed1ab_0
  python             conda-forge/linux-64::python-3.10.8-h4a9ceb5_0_cpython
  readline           conda-forge/linux-64::readline-8.1.2-h0f457ee_0
  setuptools         conda-forge/noarch::setuptools-65.5.1-pyhd8ed1ab_0
  tk                 conda-forge/linux-64::tk-8.6.12-h27826a3_0
  tzdata             conda-forge/noarch::tzdata-2022g-h191b570_0
  wheel              conda-forge/noarch::wheel-0.38.4-pyhd8ed1ab_0
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate python310forge
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Then

$ conda activate python310forge
$ conda info

     active environment : python310forge
    active env location : /home/lev/.conda/envs/cent7/2020.11-py38/python310forge
            shell level : 1
       user config file : /home/lev/.condarc
 populated config files : /home/lev/.condarc
          conda version : 4.9.2
    conda-build version : 3.20.5
         python version : 3.8.5.final.0
       virtual packages : __glibc=2.17=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /apps/spack/bell/apps/anaconda/2020.11-py38-gcc-4.8.5-nhzhrm2  (read only)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /apps/spack/bell/apps/anaconda/2020.11-py38-gcc-4.8.5-nhzhrm2/pkgs
                          /home/lev/.conda/pkgs
       envs directories : /home/lev/.conda/envs/cent7/2020.11-py38
                          /home/lev/.conda/envs
                          /apps/spack/bell/apps/anaconda/2020.11-py38-gcc-4.8.5-nhzhrm2/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Linux/3.10.0-1160.76.1.el7.x86_64 centos/7.9.2009 glibc/2.17
                UID:GID : 27004:6751
             netrc file : None
           offline mode : False

$ conda list
# packages in environment at /home/lev/.conda/envs/cent7/2020.11-py38/python310forge:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
conda-pack                0.7.0              pyh6c4a22f_0    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
openssl                   3.0.7                h0b41bf4_1    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
python                    3.10.8          h4a9ceb5_0_cpython    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
setuptools                65.5.1             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge

And

$ conda-pack --version
conda-pack 0.7.0

$ conda-pack
Collecting packages...
CondaPackError: 
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- conda-pack 0.7.0:
    lib/python3.1/site-packages/conda_pack-0.7.0.dist-info/INSTALLER
    lib/python3.1/site-packages/conda_pack-0.7.0.dist-info/LICENSE.txt
    lib/python3.1/site-packages/conda_pack-0.7.0.dist-info/METADATA
    + 19 others
- wheel 0.38.4:
    lib/python3.1/site-packages/wheel-0.38.4-py3.11.egg-info/PKG-INFO
    lib/python3.1/site-packages/wheel-0.38.4-py3.11.egg-info/SOURCES.txt
    lib/python3.1/site-packages/wheel-0.38.4-py3.11.egg-info/dependency_links.txt
    + 21 others
- pip 22.3.1:
    lib/python3.1/site-packages/pip-22.3.1-py3.11.egg-info/PKG-INFO
    lib/python3.1/site-packages/pip-22.3.1-py3.11.egg-info/SOURCES.txt
    lib/python3.1/site-packages/pip-22.3.1-py3.11.egg-info/dependency_links.txt
    + 498 others
- setuptools 65.5.1:
    lib/python3.1/site-packages/_distutils_hack/__init__.py
    lib/python3.1/site-packages/_distutils_hack/override.py
    lib/python3.1/site-packages/distutils-precedence.pth
    + 238 others

This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).

[luismarquezgft]

It works. I was running conda pack instead of conda-pack.

Thank you very much for your help.

This is the code (adapted from your example @jtilly):

conda create -y -f -n python310forge -c conda-forge python=3.10 conda-pack
conda activate python310forge
...
conda-pack

[luismarquezgft]

Hi @lgorenstein. I ran the customized Python backend in a Triton server (local docker), and it loaded successfully. Now we are trying to load it into a Kubernetes cluster.
I would like to help with the error you reported, but honestly, I don't know what is going on with that symlink or what it implies.

[lgorenstein]

@luismarquezgft sure, I totally understand. I believe this symlink handling is a glitch in conda-pack - that's why I posted it here for @jtilly to look at.

The --ignore-missing-files is a nice workaround, but as Jan said, it might just kick the can down the road. The fact that the unpacked environment does not match the original one is worrisome, and it can bite later. That's why I do not think the issue is fully resolved and propose to keep it open for Jan.

[lgorenstein]

But I am happy to spin it off as a separate issue, if that makes more sense for you, @jtilly

[jtilly]

@lgorenstein I don't think anybody will want to invest a lot of time into fixing this given that it works with more recent versions of conda. (That's correct, right?). It's my understanding that the lib/python3.1 symlink doesn't do much and can safely be removed from the environment.

So you could just run conda-pack like this:

conda-pack -f --ignore-missing-files --exclude lib/python3.1

Full example that I can run with conda=4.9.2:

conda create -y -f -n python310 -c conda-forge python=3.10 conda-pack
conda activate python310
conda-pack -f --ignore-missing-files --exclude lib/python3.1
mkdir -p out
tar -xf python310.tar.gz -C out

[morotti]

for anybody who is facing the same issue, you are using an old conda version, you need to upgrade conda.

changelog: https://conda.io/projects/continuumio-conda/en/latest/release-notes.html#id73

conda 4.11.0 (2021-11-22)

• Support Python 3.10 in version parser.

conda 4.10.0 (2021-03-30)

• Properly parse Python version >= 3.10 (#10478)

[jkh1]

Error still there with conda 23.5.2 (tested with the continuumio/miniconda3:23.5.2 docker container).
If lib/python3.1 can safely be excluded, why not simply remove it? It would also be useful to mention the issue and the workaround in the documentation.