rhodiumgroup / helm-chart Goto Github PK

@jgerardsimcock our travis deploy key is read only. I think this is the deployment problem.

The travis build log says something about read only access to the repo when trying to deploy to gh-pages, and our key is marked as read only. It looks like you can't modify it, so you'll have to re-generate a new key with write access. I don't know how the encryption was done, so don't want to mess with a PR, but I think this may get us up and running! 😄

Add ipdb to the list of installed python packages in the next cluster version.

This would be helpful with debugging.

May be part of a larger effort to combine image creation and helm chart deployment using terraform and/or Qhub.

Also, update readme so that it points to the gitlab version of docker_images and not the deprecated GitHub version.

We bump the "limits" for large clusters, but we don't bump the "requests", so it's possible (likely?) that you'll get put on a node with another user, and if you're trying to use the full memory, you're probably going to run into issues if the other user is trying to use some of the memory they've got allocated. You may not actually be able to use more than 1/2 the node memory, since the other user will also have a request of half of the memory.

To illustrate this, I recently requested a "large" container and these are some of the diagnostics:

Showing that my requested amount is half my limit amount

We're in a simultaneous multi-cluster crash loop backoff. Fun times.

If you're coming upon this thread from the interwebs... we're running multiple z2jh-based clusters based loosely on the pangeo-data/pangeo hub. We're in various stages of trying to upgrade these clusters to be more in line with the pangeo master, but in what appears to be an unrelated turn, all our hubs went down simultaneously (user notebooks & dask clusters are still running), each reporting slightly different errors. If you find yourself in this situation... may the force be with you.

This is the stacktrace for compute-test, deployed from the attempted-upgrade branch with helm2:

[E 2019-12-08 17:22:00.354 JupyterHub app:1623]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1620, in launch_instance_async
    yield self.initialize(argv)
  File "/usr/lib/python3.6/types.py", line 204, in __next__
    return next(self.__wrapped)
  File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1358, in initialize
    self.load_config_file(self.config_file)
  File "<decorator-gen-5>", line 2, in load_config_file
  File "/usr/local/lib/python3.6/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/traitlets/config/application.py", line 598, in load_config_file
    raise_config_file_errors=self.raise_config_file_errors,
  File "/usr/local/lib/python3.6/dist-packages/traitlets/config/application.py", line 562, in _load_config_files
    config = loader.load_config()
  File "/usr/local/lib/python3.6/dist-packages/traitlets/config/loader.py", line 457, in load_config
    self._read_file_as_dict()
  File "/usr/local/lib/python3.6/dist-packages/traitlets/config/loader.py", line 489, in _read_file_as_dict
    py3compat.execfile(conf_filename, namespace)
  File "/usr/local/lib/python3.6/dist-packages/ipython_genutils/py3compat.py", line 198, in execfile
    exec(compiler(f.read(), fname, 'exec'), glob, loc)
  File "/srv/jupyterhub_config.py", line 46, in <module>
    c.KubeSpawner.singleuser_image_spec = os.environ['SINGLEUSER_IMAGE']
  File "/usr/lib/python3.6/os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: 'SINGLEUSER_IMAGE'

This error occurred trying to use the following spec:

• client helm v2.12.3
• server helm v2.14.3
• jupyterhub/k8s-hub: 0.8.2
• jupyterhub version 0.8.1, based on stacktrace above (jupyterhub/Chart.yaml suggests the jupyterhub appVersion should be 0.9.6, but the stacktrace does not match).
• traitlets version 4.3.2 (required by jupyterhub 0.8.2). Note that version 4.3.3 released Oct 3, 2019
• rhodium/notebook: c66b6d910bccff118e37534c2414569ea3b6b023

We're seeing similar, but not identical, issues on other, more updated versions of these packages.

Rebuilding the stacktrace with links to the source material

Note on finding the correct version of the python package jupyterhub/jupyterhub. v0.8.2 of jupyterhub/k8s-hub's Chart.yaml file suggests we should be using jupyterhub v0.9.6. However, the below stacktrace does not match any version >=0.9.0. There is no tagged release or pypi package version 0.8.2 of jupyterhub/jupyterhub. The only version I can find of jupyterhub/jupyterhub that matches the lines shown in the stacktrace is 0.8.1, which I've linked to here.

1. jupyterhub/[email protected]#L1358, line 1620, in launch_instance_async:
   yield self.initialize(argv)
2. File "/usr/lib/python3.6/types.py", line 204, in next
   return next(self.__wrapped)
3. jupyterhub/[email protected]#L1358, in initialize
   self.load_config_file(self.config_file)
4. File "", line 2, in load_config_file
5. traitlets/config/[email protected]#L87, in catch_config_error
   return method(app, *args, **kwargs)
6. traitlets/config/[email protected]#L598, in load_config_file
   raise_config_file_errors=self.raise_config_file_errors,
7. traitlets/config/[email protected]#L562, in _load_config_files
   config = loader.load_config()
8. traitlets/config/[email protected]#L457, in load_config
   self._read_file_as_dict()
9. traitlets/config/[email protected]#L489, in _read_file_as_dict
   py3compat.execfile(conf_filename, namespace)
10. File "/usr/local/lib/python3.6/dist-packages/ipython_genutils/py3compat.py", line 198, in execfile
    exec(compiler(f.read(), fname, 'exec'), glob, loc)
11. THIS DOES NOT MATCH THE [email protected] SOURCE: /srv/jupyterhub_config.py, line 46, in
    c.KubeSpawner.singleuser_image_spec = os.environ['SINGLEUSER_IMAGE']
12. File "/usr/lib/python3.6/os.py", line 669, in getitem
    raise KeyError(key) from None

KeyError: 'SINGLEUSER_IMAGE'

The thing is, our stacktrace at line 11 displays a line, c.KubeSpawner.singleuser_image_spec = os.environ['SINGLEUSER_IMAGE'], that is not in the github source for that version. I haven't yet been able to track down where this line comes from.

In subsequent versions of jupyterhub/k8s-hub, the jupyterhub_config.py file has been removed. The Dockerfile in that directory does call jupyterhub_config.py, meaning it must be created somewhere. A quick search of the jupyterhub/jupyterhub repo shows that that file is a configuration file that is meant to be specified by the user.

This seems to be the issue:

jupyterhub/zero-to-jupyterhub-k8s#1539