Comments (7)
mauzey1
commented on July 23, 2024
@durack1 I've tried creating environments for CMOR 3.7.2 on both my Macbook and a Linux server and didn't see any AWS packages appear. Could this be specific to mamba?
from cmor.
durack1
commented on July 23, 2024
@mauzey1 good catch, just trying conda instead to see where this stuff has crept in.
And you're right:
(base) -bash-4.2$ conda create -n cmor --dry-run -c conda-forge cmor
Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 23.3.1
latest version: 23.7.2
Please update conda by running
$ conda update -n base -c conda-forge conda
Or to minimize the number of packages updated during conda update use
conda install conda=23.7.2
## Package Plan ##
environment location: ~/mambaforge/envs/cmor
added / updated specs:
- cmor
The following packages will be downloaded:
package | build
---------------------------|-----------------
ca-certificates-2023.7.22 | hbcca054_0 146 KB conda-forge
certifi-2023.7.22 | pyhd8ed1ab_0 150 KB conda-forge
cmor-3.7.2 | py311h43681a2_1 1.1 MB conda-forge
hdf5-1.14.1 |nompi_h4f84152_100 3.5 MB conda-forge
krb5-1.21.1 | h659d440_0 1.3 MB conda-forge
libcurl-8.2.1 | hca28451_0 364 KB conda-forge
libnetcdf-4.9.2 |nompi_he09a3a9_107 829 KB conda-forge
libxml2-2.11.5 | h0d562d8_0 688 KB conda-forge
numpy-1.25.2 | py311h64a7726_0 7.8 MB conda-forge
openblas-0.3.23 |pthreads_h855a84d_0 5.3 MB conda-forge
openssl-3.1.2 | hd590300_0 2.5 MB conda-forge
pip-23.2.1 | pyhd8ed1ab_0 1.3 MB conda-forge
wheel-0.41.1 | pyhd8ed1ab_0 56 KB conda-forge
------------------------------------------------------------
Total: 25.0 MB
The following NEW packages will be INSTALLED:
_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
blosc conda-forge/linux-64::blosc-1.21.4-h0f2a231_0
bzip2 conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
c-ares conda-forge/linux-64::c-ares-1.19.1-hd590300_0
ca-certificates conda-forge/linux-64::ca-certificates-2023.7.22-hbcca054_0
...from cmor.
wolfiex
commented on July 23, 2024
Our culprit is the libnetcdf library. @mauzey1 @durack1
cmor==3.7.2
├─ libnetcdf 4.9.2 [required: >=4.9.2,<4.9.3.0a0]
│ ├─ aws-sdk-cpp 1.10.57 [required: >=1.10.57,<1.10.58.0a0]
│ │ ├─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ ├─ aws-c-event-stream 0.3.1 [required: >=0.3.1,<0.3.2.0a0]
│ │ │ ├─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ ├─ aws-c-io 0.13.28 [required: >=0.13.28,<0.13.29.0a0]
│ │ │ │ ├─ aws-c-cal 0.6.0 [required: >=0.6.0,<0.6.1.0a0]
│ │ │ │ │ ├─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ │ │ └─ openssl 3.1.2 [required: >=3.1.1,<4.0a0]
│ │ │ │ │ └─ dependencies of openssl displayed above
│ │ │ │ └─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ ├─ aws-checksums 0.1.16 [required: >=0.1.16,<0.1.17.0a0]
│ │ │ │ └─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ └─ libcxx 16.0.6 [required: >=15.0.7]
│ │ ├─ aws-crt-cpp 0.20.3 [required: >=0.20.3,<0.20.4.0a0]
│ │ │ ├─ aws-c-auth 0.7.0 [required: >=0.7.0,<0.7.1.0a0]
│ │ │ │ ├─ aws-c-cal 0.6.0 [required: >=0.6.0,<0.6.1.0a0]
│ │ │ │ │ └─ dependencies of aws-c-cal displayed above
│ │ │ │ ├─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ │ ├─ aws-c-http 0.7.11 [required: >=0.7.11,<0.7.12.0a0]
│ │ │ │ │ ├─ aws-c-cal 0.6.0 [required: >=0.6.0,<0.6.1.0a0]
│ │ │ │ │ │ └─ dependencies of aws-c-cal displayed above
│ │ │ │ │ ├─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ │ │ ├─ aws-c-compression 0.2.17 [required: >=0.2.17,<0.2.18.0a0]
│ │ │ │ │ │ └─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ │ │ └─ aws-c-io 0.13.28 [required: >=0.13.28,<0.13.29.0a0]
│ │ │ │ │ └─ dependencies of aws-c-io displayed above
│ │ │ │ ├─ aws-c-io 0.13.28 [required: >=0.13.28,<0.13.29.0a0]
│ │ │ │ │ └─ dependencies of aws-c-io displayed above
│ │ │ │ └─ aws-c-sdkutils 0.1.11 [required: >=0.1.11,<0.1.12.0a0]
│ │ │ │ └─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ ├─ aws-c-cal 0.6.0 [required: >=0.6.0,<0.6.1.0a0]
│ │ │ │ └─ dependencies of aws-c-cal displayed above
│ │ │ ├─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ ├─ aws-c-event-stream 0.3.1 [required: >=0.3.1,<0.3.2.0a0]
│ │ │ │ └─ dependencies of aws-c-event-stream displayed above
│ │ │ ├─ aws-c-http 0.7.11 [required: >=0.7.11,<0.7.12.0a0]
│ │ │ │ └─ dependencies of aws-c-http displayed above
│ │ │ ├─ aws-c-io 0.13.28 [required: >=0.13.28,<0.13.29.0a0]
│ │ │ │ └─ dependencies of aws-c-io displayed above
│ │ │ ├─ aws-c-mqtt 0.8.14 [required: >=0.8.14,<0.8.15.0a0]
│ │ │ │ ├─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ │ ├─ aws-c-http 0.7.11 [required: >=0.7.11,<0.7.12.0a0]
│ │ │ │ │ └─ dependencies of aws-c-http displayed above
│ │ │ │ └─ aws-c-io 0.13.28 [required: >=0.13.28,<0.13.29.0a0]
│ │ │ │ └─ dependencies of aws-c-io displayed above
│ │ │ ├─ aws-c-s3 0.3.13 [required: >=0.3.13,<0.3.14.0a0]
│ │ │ │ ├─ aws-c-auth 0.7.0 [required: >=0.7.0,<0.7.1.0a0]
│ │ │ │ │ └─ dependencies of aws-c-auth displayed above
│ │ │ │ ├─ aws-c-cal 0.6.0 [required: >=0.6.0,<0.6.1.0a0]
│ │ │ │ │ └─ dependencies of aws-c-cal displayed above
│ │ │ │ ├─ aws-c-common 0.8.23 [required: >=0.8.23,<0.8.24.0a0]
│ │ │ │ ├─ aws-c-http 0.7.11 [required: >=0.7.11,<0.7.12.0a0]
│ │ │ │ │ └─ dependencies of aws-c-http displayed above
│ │ │ │ ├─ aws-c-io 0.13.28 [required: >=0.13.28,<0.13.29.0a0]
│ │ │ │ │ └─ dependencies of aws-c-io displayed above
│ │ │ │ └─ aws-checksums 0.1.16 [required: >=0.1.16,<0.1.17.0a0]
│ │ │ │ └─ dependencies of aws-checksums displayed above
│ │ │ ├─ aws-c-sdkutils 0.1.11 [required: >=0.1.11,<0.1.12.0a0]
│ │ │ │ └─ dependencies of aws-c-sdkutils displayed above
│ │ │ └─ libcxx 16.0.6 [required: >=15.0.7]
│ │ ├─ libcurl 8.2.0 [required: >=8.1.2,<9.0a0]
│ │ │ └─ dependencies of libcurl displayed above
│ │ ├─ libcxx 16.0.6 [required: >=15.0.7]
│ │ ├─ libzlib 1.2.13 [required: >=1.2.13,<1.3.0a0]
│ │ └─ openssl 3.1.2 [required: >=3.1.1,<4.0a0]
│ │ └─ dependencies of openssl displayed above
│ ├─ blosc 1.21.4 [required: >=1.21.4,<2.0a0]
│ │ ├─ libcxx 16.0.6 [required: >=15.0.7]
│ │ ├─ libzlib 1.2.13 [required: >=1.2.13,<1.3.0a0]
│ │ ├─ lz4-c 1.9.4 [required: >=1.9.3,<1.10.0a0]
│ │ │ └─ libcxx 16.0.6 [required: >=14.0.6]
│ │ ├─ snappy 1.1.10 [required: >=1.1.10,<2.0a0]
│ │ │ └─ libcxx 16.0.6 [required: >=14.0.6]
│ │ └─ zstd 1.5.2 [required: >=1.5.2,<1.6.0a0]
│ │ └─ dependencies of zstd displayed above
│ ├─ bzip2 1.0.8 [required: >=1.0.8,<2.0a0]
│ ├─ hdf4 4.2.15 [required: >=4.2.15,<4.2.16.0a0]
│ │ ├─ libcxx 16.0.6 [required: >=14.0.6]
│ │ ├─ libjpeg-turbo 2.1.5.1 [required: >=2.1.5.1,<3.0a0]
│ │ └─ libzlib 1.2.13 [required: >=1.2.13,<1.3.0a0]
│ ├─ hdf5 1.14.1 [required: >=1.14.1,<1.14.2.0a0]
│ │ └─ dependencies of hdf5 displayed above
│ ├─ libaec 1.0.6 [required: >=1.0.6,<2.0a0]
│ │ └─ dependencies of libaec displayed above
│ ├─ libcurl 8.2.0 [required: >=8.1.2,<9.0a0]
│ │ └─ dependencies of libcurl displayed above
│ ├─ libcxx 16.0.6 [required: >=15.0.7]
│ ├─ libxml2 2.11.4 [required: >=2.11.4,<2.12.0a0]
│ │ ├─ icu 72.1 [required: >=72.1,<73.0a0]
│ │ ├─ libiconv 1.17 [required: >=1.17,<2.0a0]
│ │ ├─ libzlib 1.2.13 [required: >=1.2.13,<1.3.0a0]
│ │ └─ xz 5.2.6 [required: >=5.2.6,<6.0a0]
│ ├─ libzip 1.9.2 [required: >=1.9.2,<2.0a0]
│ │ ├─ bzip2 1.0.8 [required: >=1.0.8,<2.0a0]
│ │ ├─ libzlib 1.2.13 [required: >=1.2.12,<1.3.0a0]
│ │ └─ openssl 3.1.2 [required: >=3.0.5,<4.0a0]
│ │ └─ dependencies of openssl displayed above
│ ├─ libzlib 1.2.13 [required: >=1.2.13,<1.3.0a0]
│ ├─ openssl 3.1.2 [required: >=3.1.1,<4.0a0]
│ │ └─ dependencies of openssl displayed above
│ └─ zstd 1.5.2 [required: >=1.5.2,<1.6.0a0]
│ └─ dependencies of zstd displayed above
The full dependancy tree of cmor is attached below.
from cmor.
durack1
commented on July 23, 2024
@wolfiex thanks for the detective work. I recall that they have been folding zarr into netcdf, so this is likely what is going on here. I wonder if we can limit the dependencies somehow?
from cmor.
matthew-mizielinski
commented on July 23, 2024
@wolfiex, could you try this without going through mamba? I managed to build our environment through conda yesterday and it didn't include the aws-c-* dependencies. The aws-c-* stuff wasn't the only difference in the environment created (a few other odd minor version differences in other (non-cmor) dependencies -- I'm not sure how much to worry about this.
P.S. what was the magic spell that you used to produce the above dependency tree? I've struggled to work this out before
from cmor.
mauzey1
commented on July 23, 2024
libnetcdf added the AWS SDK for C++ to their conda-forge feedstock last month: conda-forge/libnetcdf-feedstock#180
from cmor.
mauzey1
commented on July 23, 2024
libnetcdf-feedstock removed the aws-sdk-cpp dependency so this should no longer be an issue.
from cmor.
Related Issues (20)
- Exposing latest netcdf 4.9.x library functionality: quantize, zstandard HOT 13
- Remove unused attributes when processing CMIP6Plus datasets HOT 14
- Exposing latest netcdf 4.9.x library functionality: quantize, zstandard HOT 48
- unclear warning... HOT 5
- bounds required on singleton lon and lat? HOT 5
- avoid attributes of bounds of auxilliary coordinates (`vertices_latitudes` / `vertices_longitude`) HOT 5
- Calibrating CMOR3 & 4 forward development plans HOT 7
- CMOR 3.8.0 Release HOT 4
- Update README.md to remove v3.7 reference
- default `realm = "REALM"` is always written although not required by CV HOT 2
- order in `required_global_attributes` matters HOT 1
- input time type as INT HOT 3
- CircleCI current image deprecated HOT 1
- Numpy 2.0 compatibility issue HOT 2
- File output size and chunking HOT 10
- Message Logging generates duplicated output lines
- DESTDIR support HOT 2
- Failure invoking set_deflate HOT 1
- user defined grid mapping
- Add CITATION.cff file HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cmor.