Bring into line with the Kubernetes operator design pattern about kubespawner

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[manics]

This all sounds really cool! I think the main question is how we go about developing and exploring this in a minimally disruptive way, given that it's a very fundamental change to KubeSpawner.

Do you think this can be developed in a separate repository, with a new spawner that subclasses KubeSpawner? We can make changes to KubeSpawner to support overriding existing methods, or parts of methods (either by splitting up existing methods or adding hooks)?

The Z2JH hub image only uses released KubeSpawner versions, this also applies to dev versions of Z2JH, so a new container image would need to be built anyway (extending quay.io/jupyterhub/k8s-hub:<TAG>).

Z2JH supports pass-through configuration of traitlets (this isn't yet fully documented jupyterhub/zero-to-jupyterhub-k8s#3271) which makes it easier to configure new traitlets. We'd need to do some work to allow the hub RBAC permissions to be configurable, or alternatively we could say initially the roles must be created separately, and the role name is configure in Z2JH.

I think overall this will lead to a faster development cycle since it minimises potentially protracted design conversations about changes to existing resources, leaves room for making multiple breaking changes in the new spawner whilst trying different approaches, and once it's working we can decide whether/how to merge everything back in.

[manics]

Also you're more than welcome to join our next Collaboration Cafe, it's open to everyone: jupyterhub/team-compass#718

[qcaas-nhs-sjt]

This all sounds really cool! I think the main question is how we go about developing and exploring this in a minimally disruptive way, given that it's a very fundamental change to KubeSpawner.

Do you think this can be developed in a separate repository, with a new spawner that subclasses KubeSpawner? We can make changes to KubeSpawner to support overriding existing methods, or parts of methods (either by splitting up existing methods or adding hooks)?

I had thought this might be one way forward, my concern is that then we end up with two code bases to maintain, so I had wondered if there was a way in which we could do it inside of the current codebase with minimal disruption, however I need to better familiarise myself with the existing code base to be sure. I had considered doing a PoC on our fork to see what we can do with it but if everyone else would prefer we could start up a new repository, do it there and accept the cost of maintaining two code bases.

If we don't mind this overhead, then I would recommend that we actually create a number of new repositories for this piece of work so that the various aspects of the system can be managed as separate microservices and libraries:

• Shared libraries - This would include the python models generated from the CRD's, this will act as an enabler and will allow others to easily extend in future, or to build their own operators using the model.
• The kubespawner
• The Jupyter Notebooks Operator

While this will initially increase complexity, it will ultimately make it easier to manage as each of the products develop

The Z2JH hub image only uses released KubeSpawner versions, this also applies to dev versions of Z2JH, so a new container image would need to be built anyway (extending quay.io/jupyterhub/k8s-hub:<TAG>).

Yes you are correct, we will of course need several new container images, for the new jupyter hub and another for the new operator

Z2JH supports pass-through configuration of traitlets (this isn't yet fully documented jupyterhub/zero-to-jupyterhub-k8s#3271) which makes it easier to configure new traitlets. We'd need to do some work to allow the hub RBAC permissions to be configurable, or alternatively we could say initially the roles must be created separately, and the role name is configure in Z2JH.

I'm happy with either model, though it should be relatively minor for us to add in a section to the z2jh templates for adding additional permissions to the existing roles. I will try to have a look at this and see if I can raise an issue on this over on that repository when I've got 5 minutes.

I think overall this will lead to a faster development cycle since it minimises potentially protracted design conversations about changes to existing resources, leaves room for making multiple breaking changes in the new spawner whilst trying different approaches, and once it's working we can decide whether/how to merge everything back in.

I agree, by keeping components split off in line with microservice development principles, it should accelerate development in the long run, though it should be noted that there will be a learning curve with the new model, so it might slow things down for a short while people get familiar with it.

Also you're more than welcome to join our next Collaboration Cafe, it's open to everyone: jupyterhub/team-compass#718

Thanks, will try to attend if I can 😄

[qcaas-nhs-sjt]

Following the collaboration cafe earlier and the comments above I'm contemplating trying to put together a PoC of this in my free time. As this will be a rewrite of the kubespawner I'm also considering trying to make some other improvements along the way.

One such idea would be to mirror the existing V1Pod schema and classes into our own custom resource definition, this would allow the user to change literally any aspect of the pod without having to implement custom code, the vast majority of changes could potentially be done in the code. In order to do this we could build new classes that inherit from the existing classes, these classes could implement a method that applies string replacements on any pertinent fields allowing us to supply variables into these fields based upon information collected by kubespawner. An example implementation might look like

apiGroup: kubespawner.jupyterhub.org/v1
kind: JupyterNotebookInstanceTemplate
metadata:
   name: default
   namespace: jupyterhub
spec:
   pods:
      - name: "{username}"
        weight: 100
        annotations:
           hub.jupyter.org/username: "{unescaped_username}"
        labels:
           app: jupyterhub
           component: singleuser-server
           hub.jupyter.org/username: "{username}"
        spec:
          affinity:
           nodeAffinity:
             preferredDuringSchedulingIgnoredDuringExecution:
             - preference:
                 matchExpressions:
                 - key: hub.jupyter.org/node-purpose
                   operator: In
                   values:
                   - user
           containers:
           - name: notebook
             image: myrepo/myimage:1.2.3
             env:
             - name: JUPTERHUB_USERNAME
               value: "{unescaped_username}"
             - name: JUPYTERHUB_OAUTH_CALLBACK_URL
               value: "/user/{unescaped_username}/oauth_callback"
...

When the new version of kubespawner creates a resource it will then reference this template as follows:

apiGroup: kubespawner.jupyterhub.org/v1
kind: JupyterNotebookInstance
metadata:
   name: joe-2ebloggs-40some-2eorg
   namespace: jupyterhub
spec:
   template:
     name: default
     namespace: jupyterhub   
   variables:
     unescaped_username: "[email protected]"
     username: "joe-2ebloggs-40some-2eorg"

The operator would in turn create a pod as follows:

apiVersion: v1
kind: Pod
metadata:
   name: joe-2ebloggs-40some-2eorg
   namespace: jupyterhub
spec:   
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
            - key: hub.jupyter.org/node-purpose
              operator: In
              values:
              - user
    containers:
    - name: notebook
      image: myrepo/myimage:1.2.3
      env:
      - name: JUPTERHUB_USERNAME
        value: "[email protected]"
      - name: JUPYTERHUB_OAUTH_CALLBACK_URL
        value: "/user/[email protected]/oauth_callback"

This methodology I think would give a huge amount of flexibility to customise the service without having to enter code itself, you could still extend the services but the needs to do so would be limited to more edge-cases.

I was wondering if anyone had any thoughts on this? or ideas of anything else that they would like to see from such a rewrite?

[manics]

I like this idea, I've previously thought about a Helm-template-spawner and this is along the same lines. This a lot easier to understand and extend.

Are you planning to restrict the new spawner to only handling CRDs of type kubespawner.jupyterhub.org, or are you thinking of making the spawner handle any templated k8s manifest, so e.g. an admin provides a directory of templates and the spawner processes them all for each user?

As mentioned in the collab cafe don't feel obliged to subclass KubeSpawner if a fresh start is easier. If we want to make this available in Z2JH as an option we can easily install both spawners in the hub image.