oceanhackweek / jupyter-image Goto Github PK

Here's a way to start analyzing what packages are causing the build to explode.

import json
from pathlib import Path
import pandas as pd

pkg_files = Path("/opt/conda/conda-meta/").glob("*.json")

paths = []

for pkg_file in pkg_files:
    with pkg_file.open() as f:
        pkg = json.load(f)
        paths += pkg["paths_data"]["paths"]

df = pd.DataFrame(paths)
df = df.drop(
    [
        "path_type",
        "sha256",
        "sha256_in_prefix",
        "no_link",
        "file_mode",
        "prefix_placeholder",
    ],
    axis=1,
)
df = df.dropna()
df = df.sort_values("size_in_bytes", ascending=False)

 $ df.head(20)

                                                    _path  size_in_bytes
37480   lib/python3.9/site-packages/tensorflow/python/...    271301744.0
25649                                    lib/libavcodec.a    152533588.0
25258                                   lib/libLLVM-11.so    105929424.0
74839   x86_64-conda-linux-gnu/sysroot/usr/lib64/local...     99188496.0
85596                                    lib/librsvg-2.so     97415432.0
85597                                  lib/librsvg-2.so.2     97415432.0
85598                             lib/librsvg-2.so.2.47.0     97415432.0
91707                                   lib/libLLVM-10.so     95685352.0
50318                        lib/libQt5WebEngineCore.so.5     92408776.0
50320                   lib/libQt5WebEngineCore.so.5.12.9     92408776.0
50317                          lib/libQt5WebEngineCore.so     92408776.0
50319                     lib/libQt5WebEngineCore.so.5.12     92408776.0
122433                                         bin/pandoc     76341040.0
25661                                   lib/libavformat.a     47442464.0
33070   site-packages/compliance_checker/tests/data/ma...     42981152.0
40321                                      lib/libgdal.so     35682192.0
40323                               lib/libgdal.so.28.0.1     35682192.0
40322                                   lib/libgdal.so.28     35682192.0
80393                                lib/libclang.so.11.1     35233816.0
80392                                     lib/libclang.so     35233816.0

Caching some of the build with #23

@ocefpaf

@myrandaGoesToSpace needs for her tutorial

• Pandas
• Xarray
• Seaborne
• Matplotlib

https://github.com/mamba-org/gator may work as a replacement for nb_conda_kernels that can work with a micromamba image to help make containers lighter weight.

@mhidas is presenting a tutorial accessing data from IMOS, and expect to need

jupyter
jupyterlab
owslib
pandas
xarray
cartopy
holoviews
hvplot
geoviews
netcdf4
s3fs
zarr
h5netcdf

If you want to try this image out locally, docker run -p "8080:8080" ghcr.io/oceanhackweek/jupyer-image jupyter lab --port=8080 --ip=0.0.0.0.

As of writing this, the image is about 5.8 GB, so it might take some time to download.

Please comment with any Python packages that are need for OHW22 tutorials here. We are looking to get all of these at least a week before the main event starts, so we aren't changing anything for the undergrads!

Users can create their own conda environments for projects, so those packages do not need to be installed into the base environment (which will help keep it smaller and faster).

Please use a checkbox for each package so we can track what has been added, ex:

- [ ] some-awesome-package, link to documentation if the conda package isn't obvious to someone who doesn't use it

The current Python environment is defined in https://github.com/oceanhackweek/jupyter-image/blob/main/py-base/environment.yml

This issue comment was the most useful for getting RStudio up and running: jupyterhub/zero-to-jupyterhub-k8s#990 (comment)

Also explored setting up a new profile:

• jupyterhub/jupyter-rsession-proxy#93
• https://github.com/riazarbi/k8s_setup/blob/master/configure_sandbox.sh#L48-L53
• jupyterhub/jupyter-rsession-proxy#73

• cmocean (mentioned during tutorials)
• movingpandas (From shiptrack group)

Please comment with any R packages that are need for OHW22 here. We are looking to get all of these at least a week before the main event starts, so we aren't changing anything for the undergrads!

Please use a checkbox for each package so we can track what has been added, ex:

- [ ] some-awesome-package, link to documentation if the conda package isn't obvious to someone who doesn't use it, or CRAN source

The current R environment is defined in https://github.com/oceanhackweek/jupyter-image/blob/main/r/environment.yml

@emiliom and @salvis2 I guess we need to tweak a name somewhere b/c we got an image built, see https://hub.docker.com/repository/registry-1.docker.io/uwhackweeks/oceanhackweek/tags but the GH actions on master failed looking for a :latest tag instead. It is late here and I'll debug this tomorrow morning.

We are using conda-lock to generate a lock file to get more deterministic and repeatable builds. When the environment.yml is updated, run conda-lock -p linux-64 -f environment.yml to regenerate the lockfile.

Ahh, we don't have to make our own script. We can add littler via Conda, then:

install.R -r https://cloud.r-project.org/ biomod2
installGithub.R BigelowLab/rasf BigelowLab/ohwobpg

Originally posted by @abkfenris in #17 (comment)

Overall tracking issue for any changes we need to make to our environment for OceanHackWeek 21.