infobloxopen / konk Goto Github PK

apiserver builder has had several API contract changes and now has support for file storage backends. Convert the example apiserver to use one of these and remove references to etcd, remove etcd pod.

https://github.com/infobloxopen/konk/blob/main/helm-charts/example-apiserver/templates/deployment.yaml#L99

non-etcd example
https://github.com/kubernetes-sigs/apiserver-builder-alpha/blob/master/example/non-etcd/cmd/apiserver/main.go#L30

The example-apiserver should have been verifying client certificates and caught this issue. Below is a openssl command to verify the certificates.

openssl verify -CAfile <(k -n default get secret -o yaml drew-apiserver-konk-proxy-client | \
   awk '/^  ca.crt/ {print $2}' | base64 -d) <(k -n default get secret -o yaml drew-apiserver-konk-proxy-client | \
   awk '/^  tls.crt/ {print $2}' | base64 -d)

The test has been failing here occasionally, even though the output looks successful:
https://github.com/infobloxopen/konk/pull/101/checks?check_run_id=1380932845

Change the kubeconfig admin phase to use the FQDN of the service. {service}.{namespace}.svc

https://github.com/infobloxopen/konk/blob/main/helm-charts/konk/scripts/provision.sh#L5

Strip down kube-apiserver to just aggregation functionalities. It may be possible to remove etcd in this situation.

Repo steps:

1. deploy apiservice
2. change apiservice registration object ie. cabundle field
3. redeploy apiservice

Observe no changes in the cluster occur

Konk uses certificate manager to sign relatively short lifespan certificates. After certificates are automatically rotated, the api service using them needs to begin using the new cert value. For cases where the service itself cannot do this itself, there is a common pattern in Kubernetes that uses a config reloader sidecar. This is a common use case and should be described in konk documentation.

The resources created by konk-service are not managed by the helm-operator, nor do they have ownerReferences. This leaves nothing to clean them up after a deletion. After deleting a KonkService, these resources remain:

Some mechanism should be used to cleanup these resources, such as refactoring so the helm-operator manages them, or adding ownerReferences so they are cleaned up by kube-controller-manager.

KONK service operator will deploy apiregistration objects pointing to the apiservice. We can verify apiservice readiness with kubectl. kubectl get apiservice tends not to be a good check of actual readiness. It will report ready even when there's no endpoints matching the service. Requesting the kind directly will verify availability ie. until kubectl get contact.example.infoblox.com; do sleep 5s; done

Helm operator then should mark the CR status ready which will let users verify their resources are deployed correctly to KONK. CR status does not currently verify pod availability, see
operator-framework/operator-sdk#4102

We need a way to verify ingress functionality running in a cluster. The approach needs to be independent of DNS records as devops does not support a declarative DNS model

The simplest way would be to start a pod in-cluster, issue calls directly to ingress controller with HOST header of our ingress route. Controller routes calls to this hostname to the correct Konk instance. Ingress object would be started with some KONK url fake or otherwise ie. konk.example.infoblox.com.

curl '--header Host: konk.example.infoblox.com' -H 'Authorization: {identity trusted creds}' http://localhost:80/apis/example.infoblox.com/v1alpha1/namespaces/default/contacts?limit=500

A/C

• ingress setup works in-cluster
• smoke test works with kubectl access or in-cluster

Provision script is creating brand new CA/Etcd CA on every start. It then walks through the k8s secrets to see if any are missing and replaces them. If it happens to replace a secret, it is most likely to break client-server communication due to CA being generated on every start.

We should manage CA as separate CertManager owned certificates. They can share an issuer, but Etcd and K8s should use different CAs. The init pod will mount these CAs and be read by kubeadm to create client/server certificates.

Kubeadm may not have flags, but has predefined file locations where the CA should be found. Mount the secret in these locations.

kubeadm init phase certs etcd-server                                                            stg-1
W1111 16:24:06.679820 2604580 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
error execution phase certs/etcd-server: couldn't load CA certificate etcd-ca: couldn't load etcd/ca certificate authority from /etc/kubernetes/pki
To see the stack trace of this error execute with --v=5 or higher

• k8s ca survives pod restarts k8s secret storage
• etcd ca survives pod restarts k8s secret storage

add a library chart to the konk repo that provides all the helpers required to add a konkservice to your chart and configure your service to use its secrets

Ref: https://helm.sh/docs/topics/library_charts/

Currently, to use konk, the service must be in the same namespace as konk and mount the konk-kubeconfig secret. Secrets cannot be mounted across namespaces, so that means the services must be in the same namespace.

Find some way to eliminate this requirement.

A few API versions need to be updated for Kubernetes v1.22 compatibility.

• networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress #233
• no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
• #248

Currently the server only includes "service", "service.svc" and "service.namespace.svc". It should also include "service.namespace.svc.cluster.local".

The secrets we bootstrap need to be owned by the KONK CR resource. When the CR is deleted, it will garbage collect the secrets.

Each KonkService should provision a kubeconfig for that service. This kubeconfig requires necessary privileges to perform aggregate API authentication and aggregate API authorization.

It may be useful to give it more general access as well. For instance, checking apiservice's inside KONK.

If you install Konk with scope=cluster and KonkService with konk.scope=namespace, the cluster does not start in obscure ways. We need to surface an error when the scope is mismatched.

One idea is instead of setting Konk values like scope in KonkService, we could read them directly from the named Konk CR.

kubeadm expires kubeconfig after a set period of time. Refresh the k8s secret containing the kubeconfig before it expires

The certs used in the kubeconfig in konk-services are maintained/renewed by cert-manager and default to a 90-day expiration. However, after using them to build the kubeconfig, konk-service never replaces them with the renewed certificate. After 90 days, the apiservice fails with certificate errors. After a cert is renewed, konk-service should rebuild and replace the kubeconfig secret.

https://infoblox.atlassian.net/browse/ATLAS-9340

X-Remote-User/Group headers are being passed through but the X-Remote-Extra-OBO is missing.

The client (kubectl, ingress, bulk) will be passing additional headers X-Remote-[User/Group/Extra-OBO]. These headers are configured in KONK to be passed along to the backend apiservice here: https://github.com/infobloxopen/konk/blob/main/helm-charts/konk/values.yaml#L29-L31

We need to verify these headers are present in the example-apiserver. It should be in a way that tests can verify it. My suspicion is that an auth plugin is the way to verify and check for these headers. https://github.com/kubernetes/sample-apiserver#authentication-plugins Anything that can intercept headers to the API Server endpoints will due here. An easy way to turn this into a test is to intercept headers and write them to log. A command line test could issue client calls and check the k8s pod logs for specific strings.

Kubectl can not be extended with custom headers. Use cURL to send custom headers and authenticate to k8s with client certificates.

k get secrets runner-konk-ingress-client -o yaml
# nginx certs can be used by cURL comand below

curl -k --cacert ca.crt -vvv -XGET  -H "User-Agent: kubectl/v1.18.8 (linux/amd64) kubernetes/9f2892a" 
-H "Accept: application/json, */*" 
-H "X-Remote-User: user-test-passed" # {unique token easy to find in logs}
'https://localhost:6443/api?timeout=32s' 2>&1

Current State
A sample auth plugin has been introduced to example-apiserver, but it does not currently receive any traffic when example-apiserver is contacted. https://github.com/infobloxopen/konk/blob/main/test/apiserver/pkg/auth/remoteheader/remoteheader.go This is one approach to intercept incoming traffic and look for headers

A/C

• PR test that verifies headers sent by a client appear in the server.

#177 (comment)

Add retry with backoff here:

The other possibility I see for improvement here would be to retry with backoff and eventually abort (exit 1) upon error. This way it would be eventually consistent during a kube-apiserver outage.

Originally posted by @kd7lxl in #120 (comment)

Adding a smoketest to konkservice could be helpful. The konkservice has all the information it needs to report whether the apiservice is healthy: the konk name, the apiservice name, and creds. You can run a command manually like this:

% k -n aggregate exec -it tagging-aggregate-api-apiservice-konk-service-kubectl-apis7ms4n -- kubectl get apiservice v1alpha1.tagging.bulk.infoblox.com
NAME                                 SERVICE                                      AVAILABLE   AGE
v1alpha1.tagging.bulk.infoblox.com   aggregate/tagging-aggregate-api-apiservice   True        45h

If something like this were to run regularly and report readiness, it could be used to drive dashboards/alerts.