Extending guide with DNS and letsencrypt about zero-to-jupyterhub-k8s

[Winterflower]

hihi @cmoscardi !
super - thanks very much for those screencaps. I think in the Google Cloud console screenshot, one of the kube-lego-nginx services is of type Ingress (Resource) and the other ClusterIP (this is for the kube-lego-nginx pod ). I think kube-lego pod creates an Ingress resource for the ACME challenge by default, and calls it (confusingly) kube-lego-nginx, which would explain why you have two entries.
I went with the default kube-lego-gce and my console also has two entries - one called kube-lego-gce (for the pod) and another to kube-lego-nginx (the type is Ingress).

Edit: I am currently hit by this issue for kube-lego jetstack/kube-lego#256

The health checks are second issue (like you say in your original post). Good news is that you can redirect them to another URL such as /healthz from the Compute Engine Health Checks screen. I think @tothandras is working on something for configurable-http-proxy. jupyterhub/configurable-http-proxy#124

[cmoscardi]

hey, sorry for the delay @Winterflower - so, I'll be back with more shortly. But yes, I explicitly changed the service to kube-lego-gce I think because of a recommendation I saw somewhere. It was nginx by default (and something really weird was happening with that, too).

Right, so with nginx, I don't see any sort of nginx backend actually turn on that would be able to serve the validation string for Let's Encrypt. In particular, I see this:

default backend - 404

So now I have to figure out setting GCE in the config.yaml file...

Well, I went down another track and discovered that I had not done a helm install for stable/nginx-ingress (as seen here).

This led me to notice the following in the list of services... The thing to note is that there are two kube-lego-nginx things running. So this is a mystery to me.

[cmoscardi]

So, after enough fiddling with it, I did get it to work! In particular, I deleted the "default https check" under the health checks page and things started working.

Notably that health check is claiming not to be used by anything? (vs. the others which say they're in use by things like k8s-be-30305--22af3a8ded6f4bb1)

How does this work? We just don't know. But I now might be able to fashion a PR out of this (for GKE people, at least).

UPDATE: I think it might have actually been two things. Sorry I'm just flying so blind here.

1. There's another health check for the jupyterhub service. I found it by matching the port in kubectl get svc to the same port on the gcloud health check page (30729 for me). I set the URL to be /hub/login. I have no idea if that's good practice, but it does seem reasonable that the login page should return a 200 if the service is up.
2. I deleted the defualt https check. I now have no idea if that did anything - I reverted back 1) to see if it was a factor, and discovered that was the case.

[yuvipanda]

Thanks for filing the issue!

I agree letsencrypt integration is the right thing to do.

[minrk]

We should definitely add letsencrypt to the walkthough!

Letsencrypt works right now, we just need to add it to the docs. FWIW, I believe this is the config necessary for letsencrypt to work in the helm chart as it is now:

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  host: your-public-host.biz
  https:
    enabled: true
    type: "kube-lego"

proxy:
  service:
    type: ClusterIP

@yuvipanda is any of that superfluous? Could we reduce the necessary config to something more like:

letsencrypt:
  domain: yourdomain.horse
  email: [email protected]

which would trigger the ingress, ClusterIP, etc. config?

[cmoscardi]

https://hackmd.io/CYNgjKDsIEwLQGMCmBWALHNsVwIYlwwGYw0ZRdIAjXADhSA= might be more up to date?

[cmoscardi]

OK. Putting this here to document, @yuvipanda would love some help before JupyterCon next week.

i'm currently running into this, I believe. the proxy service returns a 302 by default, so i think this breaks google cloud's health checking:
jetstack/kube-lego#18

You can go to https://train.thedataincubator.co/ and see that https has worked (I had to set the lego backend to be "gce", not "nginx"). However, the routing to the proxy-public service (which is also up on an external IP) is not correctly being routed through. Here's my ingress config:

spec:
  rules:
  - host: train.thedataincubator.co
    http:
      paths:
      - backend:
          serviceName: kube-lego-gce
          servicePort: 8080
        path: /.well-known/acme-challenge/*
      - backend:
          serviceName: proxy-public
          servicePort: 80
        path: /*

Long story short: I think that either / needs to return 200 or we need to fix the health check, and I'm not entirely clear how. Or I'm just doing something terribly wrong.

[Winterflower]

@cmoscardi just curious, when you ran

kubectl --namespace=<sth> get ingress

did the name of the kube-lego ingress first appear as kube-lego-nginx (as it does in my case)? Did you manually change it later to kube-lego-gce?
(context: am trying to deploy jupyterhub with kube-lego and stuck with the same issue as you)

[cmoscardi]

Right. Now, on the latest master, I'm encountering a new thing.

From chrome:

This site can’t be reached
t5.thedataincubator.co unexpectedly closed the connection.

This is with the "gce" ingress above, health check set appropriately (and apparently everything is healthy). Let's encrypt has successfully run (according to the logs). There's probably some firewall rule I need to set?

Or, an extra firewall rule being created that I needed to delete.... So that's issue #1, it seems.

Now, the next thing to happen is that upon going to https://t5.thedataincubator.co, I get redirected to /hub and get a default backend - 404 error. Here's what I see in the proxy logs:

15:26:11.695 - debug: [ConfigProxy] PROXY WEB / to http://10.27.250.79:8081

And there's nothing in the hub logs about requests inbound from the proxy. However, the standard open http:// proxy-public link works.

[cmoscardi]

As a point of good practice, I'll remark that I did get GCE ingress + https with let's encrypt working.

If you're reading this and want to also make it work, feel free to reply here and I'll get back to you ASAP with my instructions. It's a rather intricate set of steps.

It doesn't seem to be of general interest, though (there's a motion to get nginx working)

[minrk]

#229 adds support for SSL with letsencrypt with:

proxy:
  hosts:
    - <your-host>
  letsencrypt:
    contactEmail: <your-email>