"No client certificate CA names sent" about apiserver-builder-alpha

[pwittrock]

Hm, I don't know a whole lot about this. My plan has been to wait until apiserver aggregation works ootb in 1.7 and minikube is using 1.7 before digging in too deeply to the integration. However if you find a fix in the mean time would be happy to take it.

[jimmidyson]

Don't suppose you've any idea where to start looking? in kubernetes/apiserver I guess?

[pwittrock]

Hm. Maybe start by running kubectl -v and seeing if it is requesting the proper headers or what it is doing. Then I would try a different canonical version of Kubernetes (GCE using the up scripts?), and see if the apiserver works properly with that cluster. I think I have gotten this working with GCE before. Minikube runs in a vm and is setup a little differently than a standard cluster IIRC, so I would eliminate that as an issue. Let me know what information you get out of that, and I will see if I know enough to help further.

[DirectXMan12]

agreed -- I have a bit of experience with this stuff setting it up for service catalog, Heapster, custom metrics, etc.

Seeing exactly what kubectl is spitting out, as well as what the logs from the API server are, would be quite useful. If that's not helpful, I can try and make sure everything's set up correctly in the generated code.

[jimmidyson]

So here's the kube config being used by kubectl:

$ kubectl config view --minify

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/jdyson/go/src/github.com/jimmidyson/test-apiserver/apiserver.local.config/certificates/apiserver.crt
    server: https://localhost:9443
  name: test
contexts:
- context:
    cluster: test
    user: minikube
  name: test
current-context: test
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /home/jdyson/.minikube/apiserver.crt
    client-key: /home/jdyson/.minikube/apiserver.key

And the output from kubectl:

$ kubectl get myresources -v=10

I0523 10:08:20.972649   22231 loader.go:354] Config loaded from file /home/jdyson/.kube/config
I0523 10:08:20.973817   22231 cached_discovery.go:118] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/servergroups.json
I0523 10:08:20.973895   22231 cached_discovery.go:71] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/myresource.test.io/v1alpha1/serverresources.json
I0523 10:08:20.973981   22231 cached_discovery.go:118] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/servergroups.json
I0523 10:08:20.974032   22231 cached_discovery.go:71] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/myresource.test.io/v1alpha1/serverresources.json
I0523 10:08:20.974066   22231 cached_discovery.go:118] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/servergroups.json
I0523 10:08:20.974096   22231 cached_discovery.go:71] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/myresource.test.io/v1alpha1/serverresources.json
I0523 10:08:20.974155   22231 cached_discovery.go:118] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/servergroups.json
I0523 10:08:20.974188   22231 cached_discovery.go:71] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/myresource.test.io/v1alpha1/serverresources.json
I0523 10:08:20.974582   22231 round_trippers.go:398] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/v1.6.2 (linux/amd64) kubernetes/477efc3" https://localhost:9443/apis/myresource.test.io/v1alpha1/namespaces/default/myresources
I0523 10:08:20.979884   22231 round_trippers.go:417] GET https://localhost:9443/apis/myresource.test.io/v1alpha1/namespaces/default/myresources 403 Forbidden in 5 milliseconds
I0523 10:08:20.979897   22231 round_trippers.go:423] Response Headers:
I0523 10:08:20.979901   22231 round_trippers.go:426]     Content-Type: text/plain
I0523 10:08:20.979904   22231 round_trippers.go:426]     X-Content-Type-Options: nosniff
I0523 10:08:20.979907   22231 round_trippers.go:426]     Content-Length: 100
I0523 10:08:20.979910   22231 round_trippers.go:426]     Date: Tue, 23 May 2017 09:08:20 GMT
I0523 10:08:20.979927   22231 request.go:991] Response Body: User "system:anonymous" cannot list myresources.myresource.test.io in the namespace "default".
I0523 10:08:20.980027   22231 helpers.go:206] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "User \"system:anonymous\" cannot list myresources.myresource.test.io in the namespace \"default\". (get myresources.myresource.test.io)",
  "reason": "Forbidden",
  "details": {
    "group": "myresource.test.io",
    "kind": "myresources",
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "User \"system:anonymous\" cannot list myresources.myresource.test.io in the namespace \"default\"."
      }
    ]
  },
  "code": 403
}]
F0523 10:08:20.980044   22231 helpers.go:119] Error from server (Forbidden): User "system:anonymous" cannot list myresources.myresource.test.io in the namespace "default". (get myresources.myresource.test.io)

[jimmidyson]

It's the User \"system:anonymous\" bit which indicates to me that the CN from the client cert isn't being used as the username. I'm assuming that's because the client cert isn't being sent at all, as the preview openssl s_client snippet indicated.

[jimmidyson]

Found it... PR coming... The problem is here which was overwriting the client CA stuff previously set up. Just going to change the order we apply the config in https://github.com/kubernetes-incubator/apiserver-builder/blob/master/pkg/cmd/server/start.go#L151-L170

[pwittrock]

Woot. Thanks.

…

On May 23, 2017 6:55 AM, "Jimmi Dyson" ***@***.***> wrote: Found it... PR coming... The problem is here <https://github.com/kubernetes-incubator/apiserver-builder/blob/master/vendor/k8s.io/apiserver/pkg/server/options/serving.go#L229> which was overwriting the client CA stuff previously set up. Just going to change the order we apply the config in https://github.com/kubernetes- incubator/apiserver-builder/blob/master/pkg/cmd/server/start.go#L151-L170 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#37 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHhIAm16PB2e2VJJ67kIZKcU1hi4fFozks5r8uU7gaJpZM4NeZZD> .