Comments (9)
costrouc
commented on June 16, 2024
2
Wanted to link that the pyg team seems to have already been aware of this issue pyg-team/pytorch_geometric#4160, mamba-org/mamba#1542.
from conda-libmamba-solver.
wolfv
commented on June 16, 2024
1
yes, basically for mamba to properly work you need to de-prioritize in the "first level".
I tried to explain it here: https://mamba.readthedocs.io/en/latest/advanced_usage/package_resolution.html#examples
Is that the issue?
from conda-libmamba-solver.
agitter
commented on June 16, 2024
1
I can still reproduce this behavior so I provided the requested output below and the conda list output I obtain when building the environment with each of the original commands.
conda info
Output
active environment : base
active env location : /medinfo/bmi775/miniconda3
shell level : 1
user config file : /home/agitter/.condarc
populated config files : /home/agitter/.condarc
conda version : 4.14.0
conda-build version : not installed
python version : 3.7.13.final.0
virtual packages : __linux=3.10.0=0
__glibc=2.17=0
__unix=0=0
__archspec=1=x86_64
base environment : /medinfo/bmi775/miniconda3 (writable)
conda av data dir : /medinfo/bmi775/miniconda3/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /medinfo/bmi775/miniconda3/pkgs
/home/agitter/.conda/pkgs
envs directories : /medinfo/bmi775/miniconda3/envs
/home/agitter/.conda/envs
platform : linux-64
user-agent : conda/4.14.0 requests/2.28.1 CPython/3.7.13 Linux/3.10.0-1160.81.1.el7.x86_64 scientific/7.9 glibc/2.17 solver/libmamba conda-libmamba-solver/22.8.1 libmambapy/0.24.0
UID:GID : 1030:1105
netrc file : None
offline mode : False
conda list -n base --show-channel-urls
Output
# packages in environment at /medinfo/bmi775/miniconda3:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
brotlipy 0.7.0 py37h27cfd23_1003 defaults
bzip2 1.0.8 h7b6447c_0 defaults
c-ares 1.18.1 h7f8727e_0 defaults
ca-certificates 2022.07.19 h06a4308_0 defaults
certifi 2022.6.15 py37h06a4308_0 defaults
cffi 1.15.1 py37h74dc2b5_0 defaults
charset-normalizer 2.0.4 pyhd3eb1b0_0 defaults
conda 4.14.0 py37h06a4308_0 defaults
conda-libmamba-solver 22.8.1 py37h06a4308_0 defaults
conda-package-handling 1.8.1 py37h7f8727e_0 defaults
cryptography 37.0.1 py37h9ce1e76_0 defaults
cytoolz 0.11.0 py37h7b6447c_0 defaults
icu 58.2 he6710b0_3 defaults
idna 3.3 pyhd3eb1b0_0 defaults
importlib-metadata 4.11.3 py37h06a4308_0 defaults
importlib_metadata 4.11.3 hd3eb1b0_0 defaults
krb5 1.19.2 hac12032_0 defaults
ld_impl_linux-64 2.38 h1181459_1 defaults
libarchive 3.5.2 h5de8990_0 defaults
libcurl 7.84.0 h91b91d3_0 defaults
libedit 3.1.20210910 h7f8727e_0 defaults
libev 4.33 h7f8727e_1 defaults
libffi 3.3 he6710b0_2 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
libmamba 0.24.0 h1566912_0 defaults
libmambapy 0.24.0 py37hd09550d_0 defaults
libnghttp2 1.46.0 hce63b2e_0 defaults
libsolv 0.7.20 h4ff587b_0 defaults
libssh2 1.10.0 h8f2d780_0 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
libxml2 2.9.14 h74e7548_0 defaults
lz4-c 1.9.3 h295c915_1 defaults
ncurses 6.3 h5eee18b_3 defaults
openssl 1.1.1q h7f8727e_0 defaults
pip 22.1.2 py37h06a4308_0 defaults
prompt-toolkit 3.0.32 pypi_0 pypi
pycosat 0.6.3 py37h27cfd23_0 defaults
pycparser 2.21 pyhd3eb1b0_0 defaults
pygments 2.13.0 pypi_0 pypi
pyopenssl 22.0.0 pyhd3eb1b0_0 defaults
pysocks 1.7.1 py37_1 defaults
python 3.7.13 h12debd9_0 defaults
radian 0.6.4 pypi_0 pypi
rchitect 0.3.39 pypi_0 pypi
readline 8.1.2 h7f8727e_1 defaults
reproc 14.2.4 h295c915_1 defaults
reproc-cpp 14.2.4 h295c915_1 defaults
requests 2.28.1 py37h06a4308_0 defaults
ruamel_yaml 0.15.100 py37h27cfd23_0 defaults
setuptools 63.4.1 py37h06a4308_0 defaults
six 1.16.0 pypi_0 pypi
sqlite 3.39.2 h5082296_0 defaults
tk 8.6.12 h1ccaba5_0 defaults
toolz 0.11.2 pyhd3eb1b0_0 defaults
tqdm 4.64.0 py37h06a4308_0 defaults
typing_extensions 4.3.0 py37h06a4308_0 defaults
urllib3 1.26.11 py37h06a4308_0 defaults
wcwidth 0.2.5 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0 defaults
xz 5.2.5 h7f8727e_1 defaults
yaml 0.2.5 h7b6447c_0 defaults
yaml-cpp 0.7.0 h295c915_1 defaults
zipp 3.8.0 py37h06a4308_0 defaults
zlib 1.2.12 h5eee18b_3 defaults
zstd 1.5.2 ha4553b6_0 defaults
conda config --show-sources
Output
==> /home/agitter/.condarc <==
envs_dirs:
- /medinfo/bmi775/miniconda3/envs
experimental_solver: libmamba
conda env create -f pyg-minimal.yml --experimental-solver libmamba
conda list -n pyg-minimal --show-channel-urls
Output
# packages in environment at /home/agitter/.conda/envs/pyg-minimal:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
blas 2.116 mkl conda-forge
blas-devel 3.9.0 16_linux64_mkl conda-forge
brotlipy 0.7.0 py39hb9d737c_1005 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2022.12.7 ha878542_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py39he91dace_3 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
cpuonly 2.0 0 pytorch
cryptography 39.0.1 py39h079d5ae_0 conda-forge
cudatoolkit 11.6.0 hecad31d_11 conda-forge
icu 70.1 h27087fc_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
libblas 3.9.0 16_linux64_mkl conda-forge
libcblas 3.9.0 16_linux64_mkl conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libhwloc 2.8.0 h32351e8_1 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
liblapack 3.9.0 16_linux64_mkl conda-forge
liblapacke 3.9.0 16_linux64_mkl conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libsqlite 3.40.0 h753d276_0 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libxml2 2.10.3 h7463322_0 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
llvm-openmp 15.0.7 h0cdce71_0 conda-forge
markupsafe 2.1.2 py39h72bdee0_0 conda-forge
mkl 2022.1.0 h84fe81f_915 conda-forge
mkl-devel 2022.1.0 ha770c72_916 conda-forge
mkl-include 2022.1.0 h84fe81f_915 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
numpy 1.24.2 py39h7360e5f_0 conda-forge
openssl 3.0.8 h0b41bf4_0 conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pip 23.0 pyhd8ed1ab_0 conda-forge
pooch 1.6.0 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyg 2.1.0 py39_torch_1.12.0_cu116 pyg
pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.9.16 h2782a2a_0_cpython conda-forge
python_abi 3.9 3_cp39 conda-forge
pytorch 1.12.1 py3.9_cpu_0 pytorch
pytorch-cluster 1.6.0 py39_torch_1.12.0_cu116 pyg
pytorch-mutex 1.0 cpu pytorch
pytorch-scatter 2.1.0 py39_torch_1.12.0_cu116 pyg
pytorch-sparse 0.6.16 py39_torch_1.12.0_cu116 pyg
readline 8.1.2 h0f457ee_0 conda-forge
requests 2.28.2 pyhd8ed1ab_0 conda-forge
scikit-learn 1.2.1 py39h86b2a18_0 conda-forge
scipy 1.10.0 py39h7360e5f_2 conda-forge
setuptools 67.3.2 pyhd8ed1ab_0 conda-forge
tbb 2021.7.0 h924138e_1 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tqdm 4.64.1 pyhd8ed1ab_0 conda-forge
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzdata 2022g h191b570_0 conda-forge
urllib3 1.26.14 pyhd8ed1ab_0 conda-forge
wheel 0.38.4 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
conda env create -f pyg-minimal.yml --experimental-solver classic
conda list -n pyg-minimal --show-channel-urls
Output
# packages in environment at /home/agitter/.conda/envs/pyg-minimal:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
blas 1.0 mkl defaults
brotlipy 0.7.0 py39hb9d737c_1005 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2022.12.7 ha878542_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py39he91dace_3 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
cpuonly 2.0 0 pytorch
cryptography 39.0.1 py39h079d5ae_0 conda-forge
fftw 3.3.10 nompi_hf0379b8_106 conda-forge
icu 70.1 h27087fc_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
intel-openmp 2021.4.0 h06a4308_3561 defaults
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libhwloc 2.8.0 h32351e8_1 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libsqlite 3.40.0 h753d276_0 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libxml2 2.10.3 h7463322_0 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
llvm-openmp 15.0.7 h0cdce71_0 conda-forge
markupsafe 2.1.2 py39h72bdee0_0 conda-forge
mkl 2021.4.0 h8d4b97c_729 conda-forge
mkl-service 2.4.0 py39h7e14d7c_0 conda-forge
mkl_fft 1.3.1 py39h0c7bc48_1 conda-forge
mkl_random 1.2.2 py39hde0f152_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
numpy 1.23.5 py39h14f4228_0 defaults
numpy-base 1.23.5 py39h31eccc5_0 defaults
openssl 3.0.8 h0b41bf4_0 conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pip 23.0 pyhd8ed1ab_0 conda-forge
pooch 1.6.0 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyg 2.1.0 py39_torch_1.12.0_cpu pyg
pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.9.16 h2782a2a_0_cpython conda-forge
python_abi 3.9 3_cp39 conda-forge
pytorch 1.12.1 py3.9_cpu_0 pytorch
pytorch-cluster 1.6.0 py39_torch_1.12.0_cpu pyg
pytorch-mutex 1.0 cpu pytorch
pytorch-scatter 2.1.0 py39_torch_1.12.0_cpu pyg
pytorch-sparse 0.6.16 py39_torch_1.12.0_cpu pyg
readline 8.1.2 h0f457ee_0 conda-forge
requests 2.28.2 pyhd8ed1ab_0 conda-forge
scikit-learn 1.2.0 py39h6a678d5_0 defaults
scipy 1.10.0 py39h14f4228_0 defaults
setuptools 67.3.2 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
tbb 2021.7.0 h924138e_1 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tqdm 4.64.1 pyhd8ed1ab_0 conda-forge
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzdata 2022g h191b570_0 conda-forge
urllib3 1.26.14 pyhd8ed1ab_0 conda-forge
wheel 0.38.4 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
from conda-libmamba-solver.
jaimergp
commented on June 16, 2024
Hey @agitter, I am trying to reproduce this but I'll need more info to make sure I am doing the right tests. Can you post:
conda infoconda list -n base --show-channel-urlsconda config --show-sources
After assuming we have everything updated and we are running on linux-64, I did a dry-run and I can reproduce your results:
libmamba
$ CONDA_SUBDIR=linux-64 conda env create -f repro-env.yml --solver=libmamba --dry-run
conda-forge/linux-64 Using cache
conda-forge/noarch Using cache
pytorch/linux-64 Using cache
pytorch/noarch Using cache
pkgs/main/linux-64 Using cache
pkgs/main/noarch Using cache
pkgs/r/linux-64 Using cache
pkgs/r/noarch Using cache
pyg/noarch No change
pyg/linux-64 No change
Collect all metadata (repodata.json): done
Solving environment: done
name: pyg
channels:
- conda-forge
- pyg
- pytorch
dependencies:
- conda-forge/linux-64::_libgcc_mutex==0.1=conda_forge
- conda-forge/linux-64::ca-certificates==2022.12.7=ha878542_0
- conda-forge/linux-64::ld_impl_linux-64==2.40=h41732ed_0
- conda-forge/linux-64::libgfortran5==12.2.0=h337968e_19
- conda-forge/linux-64::libstdcxx-ng==12.2.0=h46fd767_19
- conda-forge/linux-64::mkl-include==2022.1.0=h84fe81f_915
- conda-forge/linux-64::python_abi==3.9=3_cp39
- pytorch/noarch::pytorch-mutex==1.0=cpu
- conda-forge/noarch::tzdata==2022g=h191b570_0
- pytorch/noarch::cpuonly==2.0=0
- conda-forge/linux-64::libgfortran-ng==12.2.0=h69a702a_19
- conda-forge/linux-64::_openmp_mutex==4.5=2_kmp_llvm
- conda-forge/linux-64::libgcc-ng==12.2.0=h65d4601_19
- conda-forge/linux-64::bzip2==1.0.8=h7f98852_4
- conda-forge/linux-64::cudatoolkit==11.6.0=hecad31d_11
- conda-forge/linux-64::icu==70.1=h27087fc_0
- conda-forge/linux-64::libffi==3.4.2=h7f98852_5
- conda-forge/linux-64::libiconv==1.17=h166bdaf_0
- conda-forge/linux-64::libnsl==2.0.0=h7f98852_0
- conda-forge/linux-64::libuuid==2.32.1=h7f98852_1000
- conda-forge/linux-64::libzlib==1.2.13=h166bdaf_4
- conda-forge/linux-64::ncurses==6.3=h27087fc_1
- conda-forge/linux-64::openssl==3.0.8=h0b41bf4_0
- conda-forge/linux-64::xz==5.2.6=h166bdaf_0
- conda-forge/linux-64::libsqlite==3.40.0=h753d276_0
- conda-forge/linux-64::libxml2==2.10.3=h7463322_0
- conda-forge/linux-64::llvm-openmp==15.0.7=h0cdce71_0
- conda-forge/linux-64::readline==8.1.2=h0f457ee_0
- conda-forge/linux-64::tk==8.6.12=h27826a3_0
- conda-forge/linux-64::libhwloc==2.8.0=h32351e8_1
- conda-forge/linux-64::python==3.9.16=h2782a2a_0_cpython
- conda-forge/noarch::appdirs==1.4.4=pyh9f0ad1d_0
- conda-forge/noarch::certifi==2022.12.7=pyhd8ed1ab_0
- conda-forge/noarch::charset-normalizer==2.1.1=pyhd8ed1ab_0
- conda-forge/noarch::colorama==0.4.6=pyhd8ed1ab_0
- conda-forge/noarch::idna==3.4=pyhd8ed1ab_0
- conda-forge/linux-64::markupsafe==2.1.2=py39h72bdee0_0
- conda-forge/noarch::packaging==23.0=pyhd8ed1ab_0
- conda-forge/noarch::pycparser==2.21=pyhd8ed1ab_0
- conda-forge/noarch::pyparsing==3.0.9=pyhd8ed1ab_0
- conda-forge/noarch::pysocks==1.7.1=pyha2e5f31_6
- conda-forge/noarch::setuptools==67.1.0=pyhd8ed1ab_0
- conda-forge/linux-64::tbb==2021.7.0=h924138e_1
- conda-forge/noarch::threadpoolctl==3.1.0=pyh8a188c0_0
- conda-forge/noarch::typing_extensions==4.4.0=pyha770c72_0
- conda-forge/noarch::wheel==0.38.4=pyhd8ed1ab_0
- conda-forge/linux-64::cffi==1.15.1=py39he91dace_3
- conda-forge/noarch::jinja2==3.1.2=pyhd8ed1ab_1
- conda-forge/noarch::joblib==1.2.0=pyhd8ed1ab_0
- conda-forge/linux-64::mkl==2022.1.0=h84fe81f_915
- conda-forge/noarch::pip==23.0=pyhd8ed1ab_0
- conda-forge/noarch::tqdm==4.64.1=pyhd8ed1ab_0
- conda-forge/linux-64::brotlipy==0.7.0=py39hb9d737c_1005
- conda-forge/linux-64::cryptography==39.0.1=py39h079d5ae_0
- conda-forge/linux-64::libblas==3.9.0=16_linux64_mkl
- conda-forge/linux-64::mkl-devel==2022.1.0=ha770c72_916
- conda-forge/linux-64::libcblas==3.9.0=16_linux64_mkl
- conda-forge/linux-64::liblapack==3.9.0=16_linux64_mkl
- conda-forge/noarch::pyopenssl==23.0.0=pyhd8ed1ab_0
- conda-forge/linux-64::liblapacke==3.9.0=16_linux64_mkl
- conda-forge/linux-64::numpy==1.24.2=py39h7360e5f_0
- conda-forge/noarch::urllib3==1.26.14=pyhd8ed1ab_0
- conda-forge/linux-64::blas-devel==3.9.0=16_linux64_mkl
- conda-forge/noarch::requests==2.28.2=pyhd8ed1ab_0
- conda-forge/linux-64::blas==2.116=mkl
- conda-forge/noarch::pooch==1.6.0=pyhd8ed1ab_0
- pytorch/linux-64::pytorch==1.12.1=py3.9_cpu_0
- conda-forge/linux-64::scipy==1.10.0=py39h7360e5f_2
- pyg/linux-64::pytorch-cluster==1.6.0=py39_torch_1.12.0_cu116
- pyg/linux-64::pytorch-scatter==2.1.0=py39_torch_1.12.0_cu116
- conda-forge/linux-64::scikit-learn==1.2.1=py39h86b2a18_0
- pyg/linux-64::pytorch-sparse==0.6.16=py39_torch_1.12.0_cu116
- pyg/linux-64::pyg==2.1.0=py39_torch_1.12.0_cu116classic
$ CONDA_SUBDIR=linux-64 conda env create -f repro-env.yml --solver=classic --dry-run
Collecting package metadata (repodata.json): done
Solving environment: done
name: pyg
channels:
- conda-forge
- pyg
- pytorch
dependencies:
- conda-forge/linux-64::_libgcc_mutex==0.1=conda_forge
- defaults/linux-64::blas==1.0=mkl
- conda-forge/linux-64::ca-certificates==2022.12.7=ha878542_0
- defaults/linux-64::intel-openmp==2021.4.0=h06a4308_3561
- conda-forge/linux-64::ld_impl_linux-64==2.40=h41732ed_0
- conda-forge/linux-64::libgfortran5==12.2.0=h337968e_19
- conda-forge/linux-64::libstdcxx-ng==12.2.0=h46fd767_19
- conda-forge/linux-64::python_abi==3.9=3_cp39
- pytorch/noarch::pytorch-mutex==1.0=cpu
- conda-forge/noarch::tzdata==2022g=h191b570_0
- pytorch/noarch::cpuonly==2.0=0
- conda-forge/linux-64::libgfortran-ng==12.2.0=h69a702a_19
- conda-forge/linux-64::_openmp_mutex==4.5=2_kmp_llvm
- conda-forge/linux-64::libgcc-ng==12.2.0=h65d4601_19
- conda-forge/linux-64::bzip2==1.0.8=h7f98852_4
- conda-forge/linux-64::fftw==3.3.10=nompi_hf0379b8_106
- conda-forge/linux-64::icu==70.1=h27087fc_0
- conda-forge/linux-64::libffi==3.4.2=h7f98852_5
- conda-forge/linux-64::libiconv==1.17=h166bdaf_0
- conda-forge/linux-64::libnsl==2.0.0=h7f98852_0
- conda-forge/linux-64::libuuid==2.32.1=h7f98852_1000
- conda-forge/linux-64::libzlib==1.2.13=h166bdaf_4
- conda-forge/linux-64::ncurses==6.3=h27087fc_1
- conda-forge/linux-64::openssl==3.0.8=h0b41bf4_0
- conda-forge/linux-64::xz==5.2.6=h166bdaf_0
- conda-forge/linux-64::libsqlite==3.40.0=h753d276_0
- conda-forge/linux-64::libxml2==2.10.3=h7463322_0
- conda-forge/linux-64::llvm-openmp==15.0.7=h0cdce71_0
- conda-forge/linux-64::readline==8.1.2=h0f457ee_0
- conda-forge/linux-64::tk==8.6.12=h27826a3_0
- conda-forge/linux-64::libhwloc==2.8.0=h32351e8_1
- conda-forge/linux-64::python==3.9.16=h2782a2a_0_cpython
- conda-forge/noarch::appdirs==1.4.4=pyh9f0ad1d_0
- conda-forge/noarch::certifi==2022.12.7=pyhd8ed1ab_0
- conda-forge/noarch::charset-normalizer==2.1.1=pyhd8ed1ab_0
- conda-forge/noarch::colorama==0.4.6=pyhd8ed1ab_0
- conda-forge/noarch::idna==3.4=pyhd8ed1ab_0
- conda-forge/linux-64::markupsafe==2.1.2=py39h72bdee0_0
- conda-forge/noarch::packaging==23.0=pyhd8ed1ab_0
- conda-forge/noarch::pycparser==2.21=pyhd8ed1ab_0
- conda-forge/noarch::pyparsing==3.0.9=pyhd8ed1ab_0
- conda-forge/noarch::pysocks==1.7.1=pyha2e5f31_6
- conda-forge/noarch::setuptools==67.1.0=pyhd8ed1ab_0
- conda-forge/noarch::six==1.16.0=pyh6c4a22f_0
- conda-forge/linux-64::tbb==2021.7.0=h924138e_1
- conda-forge/noarch::threadpoolctl==3.1.0=pyh8a188c0_0
- conda-forge/noarch::typing_extensions==4.4.0=pyha770c72_0
- conda-forge/noarch::wheel==0.38.4=pyhd8ed1ab_0
- conda-forge/linux-64::cffi==1.15.1=py39he91dace_3
- conda-forge/noarch::jinja2==3.1.2=pyhd8ed1ab_1
- conda-forge/noarch::joblib==1.2.0=pyhd8ed1ab_0
- conda-forge/linux-64::mkl==2021.4.0=h8d4b97c_729
- conda-forge/noarch::pip==23.0=pyhd8ed1ab_0
- conda-forge/noarch::tqdm==4.64.1=pyhd8ed1ab_0
- conda-forge/linux-64::brotlipy==0.7.0=py39hb9d737c_1005
- conda-forge/linux-64::cryptography==39.0.1=py39h079d5ae_0
- conda-forge/linux-64::mkl-service==2.4.0=py39h7e14d7c_0
- pytorch/linux-64::pytorch==1.12.1=py3.9_cpu_0
- defaults/linux-64::numpy-base==1.23.5=py39h31eccc5_0
- conda-forge/noarch::pyopenssl==23.0.0=pyhd8ed1ab_0
- pyg/linux-64::pytorch-cluster==1.6.0=py39_torch_1.12.0_cpu
- pyg/linux-64::pytorch-scatter==2.1.0=py39_torch_1.12.0_cpu
- conda-forge/noarch::urllib3==1.26.14=pyhd8ed1ab_0
- conda-forge/noarch::requests==2.28.2=pyhd8ed1ab_0
- conda-forge/noarch::pooch==1.6.0=pyhd8ed1ab_0
- conda-forge/linux-64::mkl_fft==1.3.1=py39h0c7bc48_1
- conda-forge/linux-64::mkl_random==1.2.2=py39hde0f152_0
- defaults/linux-64::numpy==1.23.5=py39h14f4228_0
- defaults/linux-64::scipy==1.10.0=py39h14f4228_0
- pyg/linux-64::pytorch-sparse==0.6.16=py39_torch_1.12.0_cpu
- defaults/linux-64::scikit-learn==1.2.0=py39h6a678d5_0
- pyg/linux-64::pyg==2.1.0=py39_torch_1.12.0_cpuTo note:
cpuonlycorrectly pullspytorch-mutex = cpupytorch 1.12.1frompytorchchannel uses thecpuvariant too- However, libmamba chooses the CUDA variants in the packages coming from
pyg.
One thing I've noticed in the metadata is that the pyg channel uses the cpuonly dependency as the CPU mark, while the pytorch one uses pytorch-mutex * cpu. I'd say pytorch is following the best practices here, not relying on transient dependencies.
That said, micromamba 1.3 also suffers from this issue, so there's a chance we need to fix this upstream.
micromamba
$ CONDA_SUBDIR=linux-64 ~/.local/anaconda/bin/micromamba create -f repro-env.yml --dry-run --channel-priority flexible
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
conda-forge/linux-64 Using cache
conda-forge/noarch Using cache
pytorch/linux-64 Using cache
pytorch/noarch Using cache
pyg/linux-64 No change
pyg/noarch No change
warning libmamba linux version not found (virtual package skipped)
warning libmamba glibc version not found (virtual package skipped)
Transaction
Prefix: /Users/jrodriguez/micromamba/envs/pyg
Updating specs:
- cpuonly
- pyg=2.1.0
- python=3.9
- pytorch::pytorch=1.12
warning libmamba Extracted package cache '/Users/jrodriguez/micromamba/pkgs/python_abi-3.9-3_cp39' has invalid size
warning libmamba Extracted package cache '/Users/jrodriguez/micromamba/pkgs/python_abi-3.9-3_cp39' has invalid MD5 checksum
warning libmamba Package tarball '/Users/jrodriguez/micromamba/pkgs/python_abi-3.9-3_cp39.conda' is invalid
Package Version Build Channel Size
─────────────────────────────────────────────────────────────────────────────────────────────
Install:
─────────────────────────────────────────────────────────────────────────────────────────────
+ _libgcc_mutex 0.1 conda_forge conda-forge/linux-64 3kB
+ _openmp_mutex 4.5 2_kmp_llvm conda-forge/linux-64 6kB
+ appdirs 1.4.4 pyh9f0ad1d_0 conda-forge/noarch 13kB
+ blas 2.116 mkl conda-forge/linux-64 13kB
+ blas-devel 3.9.0 16_linux64_mkl conda-forge/linux-64 13kB
+ brotlipy 0.7.0 py39hb9d737c_1005 conda-forge/linux-64 351kB
+ bzip2 1.0.8 h7f98852_4 conda-forge/linux-64 496kB
+ ca-certificates 2022.12.7 ha878542_0 conda-forge/linux-64 146kB
+ certifi 2022.12.7 pyhd8ed1ab_0 conda-forge/noarch Cached
+ cffi 1.15.1 py39he91dace_3 conda-forge/linux-64 235kB
+ charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge/noarch Cached
+ colorama 0.4.6 pyhd8ed1ab_0 conda-forge/noarch 25kB
+ cpuonly 2.0 0 pytorch/noarch 2kB
+ cryptography 39.0.1 py39h079d5ae_0 conda-forge/linux-64 1MB
+ cudatoolkit 10.2.89 h713d32c_11 conda-forge/linux-64 333MB
+ icu 70.1 h27087fc_0 conda-forge/linux-64 14MB
+ idna 3.4 pyhd8ed1ab_0 conda-forge/noarch Cached
+ jinja2 3.1.2 pyhd8ed1ab_1 conda-forge/noarch 101kB
+ joblib 1.2.0 pyhd8ed1ab_0 conda-forge/noarch 210kB
+ ld_impl_linux-64 2.40 h41732ed_0 conda-forge/linux-64 705kB
+ libblas 3.9.0 16_linux64_mkl conda-forge/linux-64 13kB
+ libcblas 3.9.0 16_linux64_mkl conda-forge/linux-64 13kB
+ libffi 3.4.2 h7f98852_5 conda-forge/linux-64 58kB
+ libgcc-ng 12.2.0 h65d4601_19 conda-forge/linux-64 954kB
+ libgfortran-ng 12.2.0 h69a702a_19 conda-forge/linux-64 23kB
+ libgfortran5 12.2.0 h337968e_19 conda-forge/linux-64 2MB
+ libhwloc 2.8.0 h32351e8_1 conda-forge/linux-64 3MB
+ libiconv 1.17 h166bdaf_0 conda-forge/linux-64 1MB
+ liblapack 3.9.0 16_linux64_mkl conda-forge/linux-64 13kB
+ liblapacke 3.9.0 16_linux64_mkl conda-forge/linux-64 13kB
+ libnsl 2.0.0 h7f98852_0 conda-forge/linux-64 31kB
+ libsqlite 3.40.0 h753d276_0 conda-forge/linux-64 810kB
+ libstdcxx-ng 12.2.0 h46fd767_19 conda-forge/linux-64 4MB
+ libuuid 2.32.1 h7f98852_1000 conda-forge/linux-64 28kB
+ libxml2 2.10.3 h7463322_0 conda-forge/linux-64 773kB
+ libzlib 1.2.13 h166bdaf_4 conda-forge/linux-64 66kB
+ llvm-openmp 15.0.7 h0cdce71_0 conda-forge/linux-64 3MB
+ markupsafe 2.1.2 py39h72bdee0_0 conda-forge/linux-64 23kB
+ mkl 2022.1.0 h84fe81f_915 conda-forge/linux-64 209MB
+ mkl-devel 2022.1.0 ha770c72_916 conda-forge/linux-64 26kB
+ mkl-include 2022.1.0 h84fe81f_915 conda-forge/linux-64 763kB
+ ncurses 6.3 h27087fc_1 conda-forge/linux-64 1MB
+ numpy 1.24.2 py39h7360e5f_0 conda-forge/linux-64 7MB
+ openssl 3.0.8 h0b41bf4_0 conda-forge/linux-64 3MB
+ packaging 23.0 pyhd8ed1ab_0 conda-forge/noarch 41kB
+ pooch 1.6.0 pyhd8ed1ab_0 conda-forge/noarch 45kB
+ pycparser 2.21 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pyg 2.1.0 py39_torch_1.12.0_cu102 pyg/linux-64 849kB
+ pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge/noarch 81kB
+ pysocks 1.7.1 pyha2e5f31_6 conda-forge/noarch Cached
+ python 3.9.16 h2782a2a_0_cpython conda-forge/linux-64 24MB
+ python_abi 3.9 3_cp39 conda-forge/linux-64 6kB
+ pytorch 1.12.1 py3.9_cpu_0 pytorch/linux-64 80MB
+ pytorch-cluster 1.6.0 py39_torch_1.12.0_cu102 pyg/linux-64 1MB
+ pytorch-mutex 1.0 cpu pytorch/noarch 3kB
+ pytorch-scatter 2.1.0 py39_torch_1.12.0_cu102 pyg/linux-64 5MB
+ pytorch-sparse 0.6.16 py39_torch_1.12.0_cu102 pyg/linux-64 2MB
+ readline 8.1.2 h0f457ee_0 conda-forge/linux-64 298kB
+ requests 2.28.2 pyhd8ed1ab_0 conda-forge/noarch 57kB
+ scikit-learn 1.2.1 py39h86b2a18_0 conda-forge/linux-64 8MB
+ scipy 1.10.0 py39h7360e5f_2 conda-forge/linux-64 24MB
+ setuptools 67.1.0 pyhd8ed1ab_0 conda-forge/noarch 576kB
+ tbb 2021.7.0 h924138e_1 conda-forge/linux-64 2MB
+ threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge/noarch 18kB
+ tk 8.6.12 h27826a3_0 conda-forge/linux-64 3MB
+ tqdm 4.64.1 pyhd8ed1ab_0 conda-forge/noarch 83kB
+ typing_extensions 4.4.0 pyha770c72_0 conda-forge/noarch 30kB
+ tzdata 2022g h191b570_0 conda-forge/noarch Cached
+ urllib3 1.26.14 pyhd8ed1ab_0 conda-forge/noarch Cached
+ xz 5.2.6 h166bdaf_0 conda-forge/linux-64 418kB
Summary:
Install: 71 packages
Total download: 741MB
─────────────────────────────────────────────────────────────────────────────────────────────
Dry run. Not executing the transaction.from conda-libmamba-solver.
agitter
commented on June 16, 2024
Can you post...
I can check it next week. This happened on a shared server I don't use often, so I'll need to confirm I can reproduce it myself on that system.
from conda-libmamba-solver.
costrouc
commented on June 16, 2024
So I'm trying to dive into this issue deeper and understand why this decision is being made by each solver.
Using the example @jaimergp showed
...
- pytorch/linux-64::pytorch==1.12.1=py3.9_cpu_0
...
- pyg/linux-64::pytorch-cluster==1.6.0=py39_torch_1.12.0_cu116
- pyg/linux-64::pytorch-scatter==2.1.0=py39_torch_1.12.0_cu116
...
- pyg/linux-64::pytorch-sparse==0.6.16=py39_torch_1.12.0_cu116
- pyg/linux-64::pyg==2.1.0=py39_torch_1.12.0_cu116
....If I dive into the depends for each package:
For pytorch builds
for a cuda pytorch build -> ('typing_extensions', 'mkl >=2018', 'python >=3.7,<3.8.0a0', 'blas * mkl', 'pytorch-mutex 1.0 cuda', 'cudatoolkit >=10.2,<10.3'))
for a cpu pytorch build -> ('typing_extensions', 'mkl >=2018', 'blas * mkl', 'python >=3.9,<3.10.0a0', 'pytorch-mutex 1.0 cpu'))
For pyg builds
('pyg', 'py38_torch_1.12.0_cu102', '2.1.0', ('pytorch-scatter', 'pytorch-sparse', 'pytorch-cluster', 'scipy', 'jinja2', 'numpy', 'pyparsing', 'requests', 'scikit-learn', 'tqdm', 'python >=3.8,<3.9.0a0', 'cudatoolkit 10.2.*', 'pytorch 1.12.*'))
('pyg', 'py37_torch_1.11.0_cpu', '2.1.0', ('pytorch-scatter', 'pytorch-sparse', 'pytorch-cluster', 'scipy', 'jinja2', 'numpy', 'pyparsing', 'requests', 'scikit-learn', 'tqdm', 'cpuonly', 'python >=3.7,<3.8.0a0', 'pytorch 1.11.*')
So the differences like we mentioned earlier is that pytorch uses pytorch-mutex 1.0 cpu and ptorch-mutex 1.0 cuda While pyg uses no indicator for the gpu builds and cpuonly for the cpu builds.
If I add pytorch-mutex==1.0=cpu as a dependency it does not help here. Along with setting explicitely - pyg==2.1.0=py39_torch_1.12.0_cpu it still finds cuda dependencies.
from conda-libmamba-solver.
jaimergp
commented on June 16, 2024
This issue can be explained with the current libsolv behavior, as hinted by @wolfv above. From the mamba docs:
This documentation talks about “track_feature”. Track features are a “deprecated” conda feature that are still used solely to “down-weight” variant packages. Conda tries to “globally minimize” the amount of track_features in an environment. In mamba, we don’t implement the same global optimization but we do de-prioritize track features. If you want to properly de-prioritize packages with mamba, please make sure to add the track_feature in the variant package or at least as a first-order dependency (e.g. numpy-1.20-pypy should either have a track feature or a specific dependency to python 3.8 *pypy which then in turn should have the track_feature).
The pytorch-geometric packages are relying on the global optimization aspects of classic, and I'd argue the recipes are under-specified. cpuonly does not work as a transitive mutex in libmamba and won't propagate / cascade. Ideally, the mutex needs to be specified in each package having GPU/CPU variants. The CPU variant needs to have pytorch-mutex * cpu, and the CUDA variant needs to have pytorch-mutex * cuda.
I patched the repodata locally and then passed it as a local channel to prove this point:
set -euxo pipefail
for subdir in linux-64 noarch; do
mkdir -p $subdir
cd $subdir
curl -sL https://conda.anaconda.org/pyg/$subdir/repodata.json -o repodata.json.orig
cd ..
done
python <<EOF
import json
import os
for subdir in ["linux-64", "noarch"]:
with open(os.path.join(subdir, "repodata.json.orig")) as f:
data = json.load(f)
for filename, record in data["packages"].items():
if "_cpu." in filename:
deps = [dep for dep in record["depends"] if dep != "cpuonly"]
deps.append("pytorch-mutex * cpu")
record["depends"] = deps
if any(dep.startswith("cudatoolkit") for dep in record["depends"]):
record["depends"].append("pytorch-mutex * cuda")
with open(os.path.join(subdir, "repodata.json"), "w") as f:
json.dump(data, f, indent=2)
EOF
cat <<EOF > pyg-minimal.yml
name: pyg
channels:
- conda-forge
- $(pwd) # pyg
- pytorch
dependencies:
- cpuonly
- pyg=2.1.0
- python=3.9
- pytorch::pytorch=1.12
EOF
CONDA_SUBDIR=linux-64 conda env create --solver libmamba --dry-run -f ./pyg-minimal.ymlSo... what can we do now?
Option A
Follow the above mutex recommendations in all the recipes submitted to the pyg channel. This can't be patched for existing packages, but new ones should amend this limitation. I don't think Anaconda.org can repodata patch stuff if they are not on the CDN infra.
For example, this block should be more like:
run:
- python {{ environ.get('PYTHON_VERSION') }}
- {{ environ.get('CONDA_PYTORCH_CONSTRAINT') }} # pytorch XXX *_cpu OR pytorch XXX *_cu*
- {{ environ.get('CONDA_PYTORCH_MUTEX') }} # pytorch-mutex * cpu OR pytorch-mutex * cuda
- {{ environ.get('CONDA_CUDATOOLKIT_CONSTRAINT') }} # cudatoolkit XXX OR empty
- pytorch-scatter * {{ environ.get('CONDA_VARIANT_GLOB') }} # *_cpu* or *_cu*
- pytorch-sparse * {{ environ.get('CONDA_VARIANT_GLOB') }} # *_cpu* or *_cu*
- pytorch-cluster * {{ environ.get('CONDA_VARIANT_GLOB') }} # *_cpu* or *_cu*
- psutil
- tqdm
- jinja2
- pyparsing
- numpy
- scipy
- requests
- scikit-learnOption B
Document this limitation for pyg users. They have to be explicit about each package with a variant, sadly. So instead of saying:
name: pyg
channels:
- conda-forge
- $(pwd) # pyg
- pytorch
dependencies:
- cpuonly
- pyg=2.1.0
- python=3.9
- pytorch::pytorch=1.12They need
name: pyg
channels:
- conda-forge
- $(pwd) # pyg
- pytorch
dependencies:
- python=3.9
- pyg=2.1.0=*_cpu*
- pytorch::pytorch=1.12=*_cpu*
- pytorch-scatter=*=*_cpu*
- pytorch-cluster=*=*_cpu*
- pytorch-sparse=*=*_cpu*Option C
Instrument libsolv to go deeper in track_features analysis so it mimics global optimization behaviour, but I don't know how tricky this is or if it's even the right approach.
from conda-libmamba-solver.
agitter
commented on June 16, 2024
Thanks @jaimergp. The detailed docs and tests in #197 are impressive.
from conda-libmamba-solver.
jaimergp
commented on June 16, 2024
Oops, I forgot to add a closing comment here 😬 Thanks for the kind words @agitter. We decided to close not because it's "solved", but because it's documented (in the docs, and also in the "Known solver issues" issue) and we can't do much at the moment other than suggesting a different packaging approach.
There's a chance the libmamba integrations in libsolv address this in the future, and in that case we will know because the included tests will fail. If that happens, I'll report the new findings here!
from conda-libmamba-solver.
Related Issues (20)
- Scheduled tests failed HOT 1
- libmamba solver doesn't match when uninstalling with wildcard HOT 2
- Scheduled tests failed HOT 1
- [DX] Update `bashrc.sh` to be compatible with libmamba v2
- Scheduled tests failed HOT 1
- conda `fetch_latest_path` interface may still read repodata into memory HOT 2
- PluginError: Error while loading conda plugins from entrypoints: libarchive.so.19: cannot open shared object file: No such file or directory HOT 1
- `conda update --all` is not idempotent HOT 1
- Scheduled tests failed HOT 1
- `conda update --all` is not idempotent HOT 7
- Support for build_number in MatchSpec HOT 6
- PackagesNotFoundError listing more than missing package HOT 8
- Installing packages in different order gives totally different versions HOT 4
- Scheduled tests failed
- Error "No module named pip" with Python2.7 HOT 4
- libmamba picks a build for the wrong Python version
- i have issue installing tensorflow-gpu
- i have issue with WSL HOT 1
- CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic HOT 1
- Conda update flips between two versions of the same package from the same channel HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conda-libmamba-solver.