Support mounting arbitrary Kubernetes secrets into single-user notebook pods about kubespawner

[liyinan926]

@betatim @yuvipanda I cloned the doc under my personal google account so it's easier to share it publicly. New doc at https://docs.google.com/document/d/1oVqZnbs4u57DxBuTqc9TD40pOM1FSpBOFlV3BW5j8Fo/edit?usp=sharing.

[consideRatio]

I don't see a clear feature suggestion, and since the linked document isn't readable by me, I'll close this.

For users that wants to mount secrets individual to the users, I would suggest the mechanism of configuring c.KubeSpawner.pre_spawn_hook to a function that dynamically configures the user-individual spawner object based on properties of the user, for example then updating the user individual spawners volumes and volume_mounts to mount a k8s secret for that user-pod specifically.

[yuvipanda]

Awesome, re: design doc! Thanks, @liyinan926!

[yuvipanda]

This was the doc from @liyinan926 - https://docs.google.com/document/d/1DDm7N2sZ4wSHdCzzcsHrtp8rvOttGuMn7MadxScvfnI/edit?ts=5a00b644#heading=h.14lxv2es4gqw

I'm at a conference until wednesday unfortunately! I'll look through it as soon as I can!

[liyinan926]

@yuvipanda just wondering have you got a chance to look at the doc?

[betatim]

I can't read the document :( Would be good to adjust the permissions so it can be read by anyone without having to request permissions.

[yuvipanda]

@liyinan926 I have read it and have Opinions(tm) :) I'll respond in more detail later today.

Can you open the doc up to public viewing / commenting?

[yuvipanda]

@liyinan926 I didn't make it today, sorry! Shall tomorrow afternoon!

[yuvipanda]

@liyinan926 thank you for your patience and your hard work in making the design doc! We highly appreciate your help!

I've been thinking about the configuration practices for JupyterHub in general and KubeSpawner in particular, and I think here are some general principles we should try hold to:

1. All config should be in the form of traitlets only. This allows for easy testing & is consistent with the whole of the JupyterHub ecosystem. The current value of all the configurable traitlets of a spawner should fully capture its configuration, without requiring references to external sources (like configmaps)
2. We should make common configuration cases easy
3. We should make complex, advanced configuration cases possible by depending on the fact that jupyterhub_config.py file is Python code, and you as an admin can actually write code here. You can subclass things, pass in callables, read from the filesystem, etc.

These haven't really been written down anywhere else, so thank you for starting this conversation in such a design centric way!

I've tried to undersand what you had intended to convey with the design doc, and I think we can make a nice solution that sticks to the traitlets design instead of mixing traitlets + configmap. In this comment, I shall outline the current amount of dynamism available, and next steps to meet the requirements I understand from the doc. Let me know if I've misunderstood anything!

Current situation

You can currently mount secrets by modifying c.KubeSpawner.volumes and c.KubeSpawner.volume_mounts. However, this is not fully static - you can use '{username}' and '{userid}' in these traitlets(https://github.com/jupyterhub/kubespawner/blob/master/kubespawner/spawner.py#L608). This enforces naming conventions on admins, but means you can already do per-user secret mounts.

New features needed

I think to achieve the full amount of features that is present in the design doc, we can do a bunch of smaller, well scoped improvements to kubespawner.

More dynamic customization than just username / userid

Just username and userid might not be enough to figure out which secrets to mount. To begin with, we might want to augment the template expansion with more information:

1. Data from options form that may be submitted pre-spawn
2. Information from auth_state (this could include what 'class' of user they are, etc)
3. ???

We could also allow callables here - although I'm somewhat loathe to do that without concrete use cases. Usually for callables we have to provide them with the spawner object, and so this means admins' config files might break badly with each version upgrade...

Environment variables from configmaps / secrets / downward API

Currently you can only specify environment variables with c.KubeSpawner.environment, and they can either be static key value pairs or 'dynamic' where the value is determined by a callable. There's no way to plug in an environment variable sourced from a ConfigMap, Secret or Downward API. We can fix this by allowing values for c.KubeSpawner.environment to be dicts themselves, thus allowing use of secrets / configmap / downward API.

Dynamic reloading of config without a restart

One of the benefits of putting config in configmap rather than as traitlets is dynamic configuration - as the configmap changes, your app's configuration also changes dyanmically without needing a restart. With a pure traitlets situation, you can not do this seamlessly right now.

I don't have a clean solution for this yet. One possibility is to make your jupyterhub_config.py read its config from a ConfigMap projected as a volume into the hub's pod (like https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/images/hub/jupyterhub_config.py#L6), and have a process supervisor restart the hub process whenever the contents of the configmap dir changes.

Out of scope

1. This is all independent of creating / deleting new namespaces.
2. It doesn't talk about namespace specific config vs user specific config. This comment assumes that what you really want is user-specific config.

Next steps?

I think the three individual items here can be implemented independently of each other simply. Perhaps that's the next step?

It's also possible I've misunderstood what the design doc was trying to accomplish, and have hence produced alternatives against a strawman. I'm happy to jump on a video call (or make a trip to the South Bay!) to clarify if that helps.

Again, thank you very much for your work, @foxish and @liyinan926! We appreciate it :) My backlog has cleared up a bit + I'm not travelling this week, so my responses should be quicker from now.

[yuvipanda]

Also /cc @minrk who is the lead dev on JupyterHub and might have opinions too.

[liyinan926]

@yuvipanda Thanks for the awesome write up! Looking forward to the new features!

[yuvipanda]

@liyinan926 do you want to pick up work on any of them? :)

[jkinkead]

I really like the idea of using a Callable for traitlets in general, since it gives control to the user over how configuration is loaded - and it can be dynamic without having to trigger restarts.

However, I understand the hesitation in passing the KubeSpawner instance as an argument, given that the API isn't necessarily fixed.

BUT: Most of these dynamic configurations are asking for per-user configuration values. The only API surface needed would be the user field; even passing that in directly (and not passing the spawner reference at all) should be sufficient for any of these. I know that access to the ID would be almost enough; and access to the ID + the decoded auth data (user groups) would be 100% sufficient.

Thoughts?

[gcavalcante8808]

Hi Folks,

Its possible to provide examples about configmap mounting for the singleuser image? I have the following code that use s3-config for singleuser image when the env variable 'USE_S3' is defined for the hub config. The config is the following:

if use_s3_storage:
  c.KubeSpawner.env_keep += [
                            "S3_ENDPOINT_URL", 
                            "S3_ACCESS_KEY",
                            "S3_SECRET_KEY",
                            "S3_BUCKET",
                            "S3_CONTENT_PREFIX"
                            ]

  #Reference S3 Configuration for use with the notebook.
  c.KubeSpawner.volumes = [
      {'name': 'notebook-config', 
       'configMap': {'name': 'jupyter-user-s3-v1',
                    'items': {'key': 'jupyter_notebook_s3_config.py',
                              'path': 'jupyter_notebook_s3_config.py'
                             }
                    }
      }
  ]

  #Define s3config as jupyter notebook config.
  c.KubeSpawner.volume_mounts = [
    {'name': 'jupyter-user-s3-v1', 
     'mountPath': '/home/jovyan/.jupyter/jupyter_notebook_config.py'
    },
  ]

I tried with this code, but it doesnt work ... So I turn on the debug statements and the following JSON was generated by the spawner when it tried to submit the request (which results in BadRequestErrror):

[E 2018-09-03 21:38:00.198 JupyterHub spawner:979] Failed for {'api_version': 'v1',
     'kind': 'Pod',
     'metadata': {'annotations': {'hub.jupyter.org/username': '01388863189'},
                  'cluster_name': None,
                  'creation_timestamp': None,
                  'deletion_grace_period_seconds': None,
                  'deletion_timestamp': None,
                  'finalizers': None,
                  'generate_name': None,
                  'generation': None,
                  'initializers': None,
                  'labels': {'app': 'jupyterhub',
                             'component': 'singleuser-server',
                             'heritage': 'jupyterhub'},
                  'name': 'jupyter-01388863189',
                  'namespace': None,
                  'owner_references': None,
                  'resource_version': None,
                  'self_link': None,
                  'uid': None},
     'spec': {'active_deadline_seconds': None,
              'affinity': None,
              'automount_service_account_token': False,
              'containers': [{'args': None,
                              'command': None,
                              'env': [{'name': 'S3_ENDPOINT_URL',
                                       'value': 'http://minio-0.default.svc.cluster.local:9000',
                                       'value_from': None},
                                      {'name': 'S3_ACCESS_KEY',
                                       'value': 'ORCUGXYV5GHE5HOGW5N5',
                                       'value_from': None},
                                      {'name': 'S3_SECRET_KEY',
                                       'value': 'XkTz2EJ068LxtnfK99E95nxgNx3DRvazZfsofnai',
                                       'value_from': None},
                                      {'name': 'S3_BUCKET',
                                       'value': 'esint',
                                       'value_from': None},
                                      {'name': 'S3_CONTENT_PREFIX',
                                       'value': 'notebooks/esint',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_API_TOKEN',
                                       'value': '9207e85361764de286eafa8fee28115d',
                                       'value_from': None},
                                      {'name': 'JPY_API_TOKEN',
                                       'value': '9207e85361764de286eafa8fee28115d',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_CLIENT_ID',
                                       'value': 'jupyterhub-user-01388863189',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_HOST',
                                       'value': '',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL',
                                       'value': '/user/01388863189/oauth_callback',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_USER',
                                       'value': '01388863189',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_API_URL',
                                       'value': 'http://10.233.14.85:8081/hub/api',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_BASE_URL',
                                       'value': '/',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_SERVICE_PREFIX',
                                       'value': '/user/01388863189/',
                                       'value_from': None}],
                              'env_from': None,
                              'image': 'hub.hom.estaleiro.serpro/ia/jupyterhub-singleuser:0.9-1',
                              'image_pull_policy': 'IfNotPresent',
                              'lifecycle': {},
                              'liveness_probe': None,
                              'name': 'notebook',
                              'ports': [{'container_port': 8888,
                                         'host_ip': None,
                                         'host_port': None,
                                         'name': 'notebook-port',
                                         'protocol': None}],
                              'readiness_probe': None,
                              'resources': {'limits': {}, 'requests': {}},
                              'security_context': None,
                              'stdin': None,
                              'stdin_once': None,
                              'termination_message_path': None,
                              'termination_message_policy': None,
                              'tty': None,
                              'volume_mounts': [{'mountPath': '/home/jovyan/.jupyter/jupyter_notebook_config.py',
                                                 'name': 'jupyter-user-s3-v1'},
                                                {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
                                                 'name': 'no-api-access-please',
                                                 'read_only': True,
                                                 'sub_path': None}],
                              'working_dir': None}],
              'dns_policy': None,
              'host_aliases': None,
              'host_ipc': None,
              'host_network': None,
              'host_pid': None,
              'hostname': None,
              'image_pull_secrets': None,
              'init_containers': [],
              'node_name': None,
              'node_selector': None,
              'restart_policy': None,
              'scheduler_name': None,
              'security_context': {'fs_group': 0,
                                   'run_as_non_root': None,
                                   'run_as_user': 0,
                                   'se_linux_options': None,
                                   'supplemental_groups': None},
              'service_account': None,
              'service_account_name': None,
              'subdomain': None,
              'termination_grace_period_seconds': None,
              'tolerations': None,
              'volumes': [{'configMap': {'items': {'key': 'jupyter_notebook_s3_config.py',
                                                   'path': 'jupyter_notebook_s3_config.py'},
                                         'name': 'jupyter-user-s3-v1'},
                           'name': 'notebook-config'},
                          {'aws_elastic_block_store': None,
                           'azure_disk': None,
                           'azure_file': None,
                           'cephfs': None,
                           'cinder': None,
                           'config_map': None,
                           'downward_api': None,
                           'empty_dir': {},
                           'fc': None,
                           'flex_volume': None,
                           'flocker': None,
                           'gce_persistent_disk': None,
                           'git_repo': None,
                           'glusterfs': None,
                           'host_path': None,
                           'iscsi': None,
                           'name': 'no-api-access-please',
                           'nfs': None,
                           'persistent_volume_claim': None,
                           'photon_persistent_disk': None,
                           'portworx_volume': None,
                           'projected': None,
                           'quobyte': None,
                           'rbd': None,
                           'scale_io': None,
                           'secret': None,
                           'storageos': None,
                           'vsphere_volume': None}]},
     'status': None}
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.6/site-packages/kubespawner/spawner.py", line 973, in start
        pod
      File "/opt/conda/lib/python3.6/concurrent/futures/thread.py", line 56, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/opt/conda/lib/python3.6/site-packages/kubespawner/spawner.py", line 942, in asynchronize
        return method(*args, **kwargs)
      File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 6561, in create_namespaced_pod
        (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
      File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 6651, in create_namespaced_pod_with_http_info
        collection_formats=collection_formats)
      File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 335, in call_api
        _preload_content, _request_timeout)
      File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
        _request_timeout=_request_timeout)
      File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 393, in request
        body=body)
      File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py", line 287, in POST
        body=body)
      File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (400)
    Reason: Bad Request
    HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Mon, 03 Sep 2018 21:37:26 GMT', 'Content-Length': '538'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Volumes: []v1.Volume: v1.Volume: VolumeSource: ConfigMap: v1.ConfigMapVolumeSource: Items: []v1.KeyToPath: ReadArrayCB: expect [ or n, but found {, error found in #10 byte of ...|\"items\": {\"key\": \"ju|..., bigger context ...|figMap\": {\"name\": \"jupyter-user-s3-v1\", \"items\": {\"key\": \"jupyter_notebook_s3_config.py\", \"path\": \"j|...","reason":"BadRequest","code":400}

So ... Do you guys have an appointment or an example on how to use configMap to help me with this?

Thanks in advance.

[gcavalcante8808]

I tried to change the code to

  c.KubeSpawner.volumes = [
      {'name': 'notebook-config', 
       'configMap': {'name': 'jupyter-user-s3-v1',
                    'items': [{'key': 'jupyter_notebook_s3_config.py',
                              'path': 'jupyter_notebook_s3_config.py'
                             }]
                    }
      }
  ]

Looks like items should be a list (of course). Corrected with the above code. After that, a small correction in the name of the volume made my day.

  c.KubeSpawner.volume_mounts = [
    {'name': 'notebook-config', 
     'mountPath': '/home/jovyan/.jupyter/jupyter_notebook_config.py'
    },
  ]

Anyway, a set of config examples would help alot. Is it a valid concern for the project? I can make some PRs with examples if it is.

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/user-specific-aws-secrets/6391/1