Comments (13)
I can reproduce your issue with conda-pack 0.6.0 when pulling it from anaconda (but I don't think there's anything we can do about that).
I can't reproduce your issue with conda-pack 0.7.0 from conda-forge.
Can you please run and report the output from:
conda create -y -f -n python310forge -c conda-forge python=3.10 conda-pack
conda activate python310forge
conda info
conda list
which conda-pack # just to make sure we're using the correct conda-pack executable
conda-pack --version
conda-pack
You can also run conda-pack with --ignore-missing-files
, which will get rid of the error message (but might just kick the can down the road).
The python3.1 symlink isn't necessarily an issue. It comes with python3.10 by default.
Edit: I somehow missed your comment about the conda versions. Yes: I can reproduce your issue with conda 4.9.2
and conda-pack 0.7.0
from conda-forge. Adding the --ignore-missing-files
flag fixes the issue for me and I can properly unpack the resulting environment without issues. So maybe try that if upgrading conda is not an option?
from conda-pack.
Also observed on our HPC cluster using older versions of Anaconda
(2020.11, conda version 4.9.2, base python version 3.8.5)
from conda-pack.
I do not think this is complete, though.
With --ignore-missing-files
, the very same python310forge
environment packed as expected:
$ conda-pack --ignore-missing-files
Collecting packages...
Packing environment at '/home/lev/.conda/envs/cent7/2020.11-py38/python310forge' to 'python310forge.tar.gz'
[########################################] | 100% Completed | 8.4s
However, I have received an error message upon unpacking:
$ module load anaconda/2020.11-py38
$ mkdir $HOME/.conda/envs/cent7/2020.11-py38/XXX
$ tar -xf python310forge.tar.gz -C $HOME/.conda/envs/cent7/2020.11-py38/XXX
tar: lib/python3.1: Cannot create symlink to `python3.10': File exists
tar: Exiting with failure status due to previous errors
The resulting environment activated and seemed to work, but the untaring error looks a bit scary, and it would definitely spook people.
Looking into the tarball, it seems that there are a lot of lines with normal lib/python3.10/....
, a line with lib/python3.1 -> python3.10
, and then a bunch of
lib/python3.1/site-packages/conda_pack-0.7.0.dist-info/INSTALLER link to lib/python3.10/site-packages/conda_pack-0.7.0.dist-info/INSTALLER
(note the symlink to a file inside python3.1 in addition to the python3.1 itself). And this is causing the error message, because upon unpacking the lib/python3.1
gets created as a directory rather than as a symlink):
$ ls -ld ~/.conda/envs/cent7/2020.11-py38/*/lib/python3.1*
lrwxrwxrwx 1 lev itap 10 Dec 13 09:27 /home/lev/.conda/envs/cent7/2020.11-py38/python310forge/lib/python3.1 -> python3.10/
drwxr-sr-x 36 lev itap 210 Dec 13 09:27 /home/lev/.conda/envs/cent7/2020.11-py38/python310forge/lib/python3.10/
drwxr-sr-x 3 lev itap 3 Dec 13 10:18 /home/lev/.conda/envs/cent7/2020.11-py38/XXX/lib/python3.1/
drwxr-sr-x 36 lev itap 210 Dec 13 10:18 /home/lev/.conda/envs/cent7/2020.11-py38/XXX/lib/python3.10/
(note how the original environment has a symlink, but the extracted one has two distinct directories)
from conda-pack.
I can't reproduce this issue. What happens when you run the following three lines?
conda create -y -n python310 python=3.10 conda-pack # (you can also use mamba instead of conda here)
conda activate python310
conda-pack
This works on Linux and macos for me.
Based on the error message that you're getting, I suspect that you ran pip install <something>
in your Python 3.10 environment before trying to run conda-pack.
For example, when I do
conda create -y -n python310pip python=3.10 conda-pack # (you can also use mamba instead of conda here)
conda activate python310pip
pip install 'pip<22.3.1'
conda-pack
I get
Collecting packages...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:
- pip 22.3.1:
lib/python3.10/site-packages/pip-22.3.1-py3.11.egg-info/PKG-INFO
lib/python3.10/site-packages/pip-22.3.1-py3.11.egg-info/SOURCES.txt
lib/python3.10/site-packages/pip-22.3.1-py3.11.egg-info/dependency_links.txt
+ 3 others
This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).
which looks a lot like your error message.
You seem to be using conda-pack=0.6.0
. Please use >=0.7.0
, otherwise you'll run into #199.
from conda-pack.
Hello Jan,
Thank you for looking at this! I did not run any pip installs, and I did exactly the three lines you said. Note: I do not have conda-forge in my channels, so using just the default channel I end up with conda-pack 0.6.0. which throwns an error:
$ cat ~/.condarc
channels:
- defaults
$ module --force purge
$ module load anaconda/2020.11-py38
$ conda create -y -n python310 python=3.10 conda-pack
[...]
conda-pack pkgs/main/noarch::conda-pack-0.6.0-pyhd3eb1b0_0
[...]
$ conda activate python310
$ conda-pack
Collecting packages...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:
- wheel 0.37.1:
lib/python3.1/site-packages/wheel-0.37.1-py3.9.egg-info/PKG-INFO
lib/python3.1/site-packages/wheel-0.37.1-py3.9.egg-info/SOURCES.txt
lib/python3.1/site-packages/wheel-0.37.1-py3.9.egg-info/dependency_links.txt
+ 20 others
- conda-pack 0.6.0:
lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/INSTALLER
lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/LICENSE.txt
lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/METADATA
+ 19 others
This is usually due to `pip` uninstalling or clobbering conda managed files,
...
As you can see, the pip warning is actually a red herring (note the lib/python3.1/
instead of lib/python3.10/
). So this probably looks like #199 you mentioned because default channel only has 0.6.0, not 0.7+. It would certainly be nice to have the most up-to-date conda-pack in the main conda channel, though.
But the problem is that I also see the same error with 0.7.0 from conda-forge:
$ conda create -y -n python310forge -c conda-forge python=3.10 conda-pack
[...]
conda-pack conda-forge/noarch::conda-pack-0.7.0-pyh6c4a22f_0
[...]
$ conda activate python310forge
$ conda-pack
Collecting packages...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:
- setuptools 65.5.1:
lib/python3.1/site-packages/_distutils_hack/__init__.py
lib/python3.1/site-packages/_distutils_hack/override.py
lib/python3.1/site-packages/distutils-precedence.pth
...
Again, there hasn't been any pip-ing, so the pip message is mistaken. And again the error messages shows lib/python3.1
instead of 3.10.
Strangely, there is indeed this weird 3.1 symlink in the site packages:
$ ls -ld ~/.conda/envs/cent7/2020.11-py38/python310forge/lib/python3.1
lrwxrwxrwx 1 lev itap 10 Dec 13 00:39 python3.1 -> python3.10/
drwxr-sr-x 36 lev itap 210 Dec 13 00:39 python3.10/
I wonder if it's an artifact of a (somehwhat old) Anaconda? I did not get the error with conda-pack 0.7.0 when I used the latest downloaded Miniconda (with conda 4.12 as opposed to 4.9.2). OTOH, both conda versions create that 3.1 symlink, so somehow conda-pack is capable to trace thing in some circumstances?
from conda-pack.
Thank you Jan. This being a centralized installation on an HPC cluster, updating its version of conda is, let's say, a process. So having a functional workaround with --ignore-missing-files
for older Anacondas is extremely useful! I did run into problem during extracting the packaged tarball - please see my next post in this thread with detailed description.
For completeness sake, here's the report the commands you requested:
$ module load anaconda/2020.11-py38
$ conda create -y -f -n python310forge -c conda-forge python=3.10 conda-pack
Collecting package metadata (current_repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.9.2
latest version: 22.11.1
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /home/lev/.conda/envs/cent7/2020.11-py38/python310forge
added / updated specs:
- conda-pack
- python=3.10
The following NEW packages will be INSTALLED:
_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
bzip2 conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
ca-certificates conda-forge/linux-64::ca-certificates-2022.12.7-ha878542_0
conda-pack conda-forge/noarch::conda-pack-0.7.0-pyh6c4a22f_0
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.39-hcc3a1bd_1
libffi conda-forge/linux-64::libffi-3.4.2-h7f98852_5
libgcc-ng conda-forge/linux-64::libgcc-ng-12.2.0-h65d4601_19
libgomp conda-forge/linux-64::libgomp-12.2.0-h65d4601_19
libnsl conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
libsqlite conda-forge/linux-64::libsqlite-3.40.0-h753d276_0
libuuid conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000
libzlib conda-forge/linux-64::libzlib-1.2.13-h166bdaf_4
ncurses conda-forge/linux-64::ncurses-6.3-h27087fc_1
openssl conda-forge/linux-64::openssl-3.0.7-h0b41bf4_1
pip conda-forge/noarch::pip-22.3.1-pyhd8ed1ab_0
python conda-forge/linux-64::python-3.10.8-h4a9ceb5_0_cpython
readline conda-forge/linux-64::readline-8.1.2-h0f457ee_0
setuptools conda-forge/noarch::setuptools-65.5.1-pyhd8ed1ab_0
tk conda-forge/linux-64::tk-8.6.12-h27826a3_0
tzdata conda-forge/noarch::tzdata-2022g-h191b570_0
wheel conda-forge/noarch::wheel-0.38.4-pyhd8ed1ab_0
xz conda-forge/linux-64::xz-5.2.6-h166bdaf_0
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate python310forge
#
# To deactivate an active environment, use
#
# $ conda deactivate
Then
$ conda activate python310forge
$ conda info
active environment : python310forge
active env location : /home/lev/.conda/envs/cent7/2020.11-py38/python310forge
shell level : 1
user config file : /home/lev/.condarc
populated config files : /home/lev/.condarc
conda version : 4.9.2
conda-build version : 3.20.5
python version : 3.8.5.final.0
virtual packages : __glibc=2.17=0
__unix=0=0
__archspec=1=x86_64
base environment : /apps/spack/bell/apps/anaconda/2020.11-py38-gcc-4.8.5-nhzhrm2 (read only)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /apps/spack/bell/apps/anaconda/2020.11-py38-gcc-4.8.5-nhzhrm2/pkgs
/home/lev/.conda/pkgs
envs directories : /home/lev/.conda/envs/cent7/2020.11-py38
/home/lev/.conda/envs
/apps/spack/bell/apps/anaconda/2020.11-py38-gcc-4.8.5-nhzhrm2/envs
platform : linux-64
user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Linux/3.10.0-1160.76.1.el7.x86_64 centos/7.9.2009 glibc/2.17
UID:GID : 27004:6751
netrc file : None
offline mode : False
$ conda list
# packages in environment at /home/lev/.conda/envs/cent7/2020.11-py38/python310forge:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2022.12.7 ha878542_0 conda-forge
conda-pack 0.7.0 pyh6c4a22f_0 conda-forge
ld_impl_linux-64 2.39 hcc3a1bd_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libsqlite 3.40.0 h753d276_0 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
openssl 3.0.7 h0b41bf4_1 conda-forge
pip 22.3.1 pyhd8ed1ab_0 conda-forge
python 3.10.8 h4a9ceb5_0_cpython conda-forge
readline 8.1.2 h0f457ee_0 conda-forge
setuptools 65.5.1 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tzdata 2022g h191b570_0 conda-forge
wheel 0.38.4 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
And
$ conda-pack --version
conda-pack 0.7.0
$ conda-pack
Collecting packages...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:
- conda-pack 0.7.0:
lib/python3.1/site-packages/conda_pack-0.7.0.dist-info/INSTALLER
lib/python3.1/site-packages/conda_pack-0.7.0.dist-info/LICENSE.txt
lib/python3.1/site-packages/conda_pack-0.7.0.dist-info/METADATA
+ 19 others
- wheel 0.38.4:
lib/python3.1/site-packages/wheel-0.38.4-py3.11.egg-info/PKG-INFO
lib/python3.1/site-packages/wheel-0.38.4-py3.11.egg-info/SOURCES.txt
lib/python3.1/site-packages/wheel-0.38.4-py3.11.egg-info/dependency_links.txt
+ 21 others
- pip 22.3.1:
lib/python3.1/site-packages/pip-22.3.1-py3.11.egg-info/PKG-INFO
lib/python3.1/site-packages/pip-22.3.1-py3.11.egg-info/SOURCES.txt
lib/python3.1/site-packages/pip-22.3.1-py3.11.egg-info/dependency_links.txt
+ 498 others
- setuptools 65.5.1:
lib/python3.1/site-packages/_distutils_hack/__init__.py
lib/python3.1/site-packages/_distutils_hack/override.py
lib/python3.1/site-packages/distutils-precedence.pth
+ 238 others
This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).
from conda-pack.
It works. I was running conda pack
instead of conda-pack
.
Thank you very much for your help.
This is the code (adapted from your example @jtilly):
conda create -y -f -n python310forge -c conda-forge python=3.10 conda-pack
conda activate python310forge
...
conda-pack
from conda-pack.
Hi @lgorenstein. I ran the customized Python backend in a Triton server (local docker), and it loaded successfully. Now we are trying to load it into a Kubernetes cluster.
I would like to help with the error you reported, but honestly, I don't know what is going on with that symlink or what it implies.
from conda-pack.
@luismarquezgft sure, I totally understand. I believe this symlink handling is a glitch in conda-pack - that's why I posted it here for @jtilly to look at.
The --ignore-missing-files
is a nice workaround, but as Jan said, it might just kick the can down the road. The fact that the unpacked environment does not match the original one is worrisome, and it can bite later. That's why I do not think the issue is fully resolved and propose to keep it open for Jan.
from conda-pack.
But I am happy to spin it off as a separate issue, if that makes more sense for you, @jtilly
from conda-pack.
@lgorenstein I don't think anybody will want to invest a lot of time into fixing this given that it works with more recent versions of conda. (That's correct, right?). It's my understanding that the lib/python3.1 symlink doesn't do much and can safely be removed from the environment.
So you could just run conda-pack like this:
conda-pack -f --ignore-missing-files --exclude lib/python3.1
Full example that I can run with conda=4.9.2:
conda create -y -f -n python310 -c conda-forge python=3.10 conda-pack
conda activate python310
conda-pack -f --ignore-missing-files --exclude lib/python3.1
mkdir -p out
tar -xf python310.tar.gz -C out
from conda-pack.
for anybody who is facing the same issue, you are using an old conda version, you need to upgrade conda.
changelog: https://conda.io/projects/continuumio-conda/en/latest/release-notes.html#id73
conda 4.11.0 (2021-11-22)
- Support Python 3.10 in version parser.
conda 4.10.0 (2021-03-30)
- Properly parse Python version >= 3.10 (#10478)
from conda-pack.
Error still there with conda 23.5.2 (tested with the continuumio/miniconda3:23.5.2 docker container).
If lib/python3.1 can safely be excluded, why not simply remove it? It would also be useful to mention the issue and the workaround in the documentation.
from conda-pack.
Related Issues (20)
- Is it needed to run `conda-unpack` with squashfs ? HOT 2
- Deactivate script is deprecated and not available on mamba environments HOT 1
- Fix environment clobbered by pip by repairing conda-meta HOT 1
- Add support for alternative compressors for mksquashfs HOT 2
- [Governance] Access to Settings in this repo HOT 3
- conda-pack documentation is outdated
- Support for incremental migration of conda environments
- Conda-Managed Files Deleted or Overwritten HOT 2
- Absorb in `constructor` HOT 8
- Unversioned python in conda-unpack script shebang
- Unpacking to path containing whitespace does not work
- unpack to another writable dir
- add a .unpacked file in the env prefix folder to indicate that the unpack script is running or done
- Write a Wiki page to describe local execution of test suite
- Question about how conda pack works, why the pack step is necessary HOT 4
- Would a refactor of conda-pack be welcome? HOT 1
- conda-pack is not reproducible HOT 1
- Multiple zip files HOT 1
- Published documentation is outdated
- trivial fix: executable files are not marked as executable correctly
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conda-pack.