Auth issues - call to /tokenreviews fails about skooner

[herbrandson]

Thanks for reporting the issue. A couple quick questions to help me track this down...

1. Are you logging in with a service token or do you have OIDC setup?
2. Using port forwarding, could you try doing a get on http://localhost:4654/openapi/v2 and sending me the response JSON (it'll be quite large).
3. Depending on the results of #2, I think I have a quick/easy fix. If so, would you be able to test an updated build? It would simply involve modifying the image in your k8dash deployment yaml to point at a different label (i can send you an example if you need).

Thanks for helping me track this down :)

[frohikey]

1. service token, thus your log line OIDC_URL: None is correct
2. unfortunately response is quite short 😳

default backend - 404

3. sure. I'm 100% ready to test whatever you need. I'm quite curious since I'm not a big fan of default dashboard. I was thinking about writing my own but you may save me a lots of work and become my hero 😁

[herbrandson]

Thanks @frohikey. I pushed an update to the herbrandson/k8dash:dev label. This includes two things

1. A slight change to the way tokens are validated
2. Some additional server logging at startup to verify which API's are available for your cluster

My suspicion is that number 1 won't solve the issue, but that number 2 will point us in the right direction. Let me know when you've had a chance to check it out.

Thanks!

[frohikey]

Here comes my report using image herbrandson/k8dash:dev

Client side:

Request URL: https://xxx.tld/apis/authorization.k8s.io/v1/selfsubjectaccessreviews
Request Method: POST
Status Code: 404

Logs:

OIDC_URL:  None
[HPM] Proxy created: /  ->  https://---.hcp.westeurope.azmk8s.io:443
Server started
Version Info:  {
    "major": "1",
    "minor": "12",
    "gitVersion": "v1.12.6",
    "gitCommit": "ab91afd7062d4240e95e51ac00a18bd58fddd365",
    "gitTreeState": "clean",
    "buildDate": "2019-02-26T12:49:28Z",
    "goVersion": "go1.10.8",
    "compiler": "gc",
    "platform": "linux/amd64"
}
Available APIs: [
    "admission.certmanager.k8s.io/v1beta1",
    "admissionregistration.k8s.io/v1beta1",
    "apiextensions.k8s.io/v1beta1",
    "apiregistration.k8s.io/v1",
    "apps/v1",
    "authentication.k8s.io/v1",
    "authorization.k8s.io/v1",
    "autoscaling/v1",
    "batch/v1",
    "certificates.k8s.io/v1beta1",
    "certmanager.k8s.io/v1alpha1",
    "coordination.k8s.io/v1beta1",
    "events.k8s.io/v1beta1",
    "extensions/v1beta1",
    "metrics.k8s.io/v1beta1",
    "monitoring.coreos.com/v1",
    "networking.k8s.io/v1",
    "policy/v1beta1",
    "rbac.authorization.k8s.io/v1",
    "scheduling.k8s.io/v1beta1",
    "storage.k8s.io/v1"
]
...
GET /oidc
...
POST /apis/authorization.k8s.io/v1/selfsubjectaccessreviews

[herbrandson]

Thanks for the info. Really sorry you're still having this issue. I'm not sure why you'd be getting a 404 on that call. The logs clearly show that the api exists. I've posted some questions about it and will let you know when I know more. Thanks for your patience.

[herbrandson]

@frohikey I've been working with a couple people from the k8s team and the general consensus is that this is happening because the provided token is failing to authenticate. Now, why that's returning a 404 seems strange. I hate to ask, but would you mind double checking that the token you're providing is valid?

[frohikey]

I'm 100% positive. I decided to test API call directly. Here's token:

Name:         k8dash-sa-token-nlx2m
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: k8dash-sa
              kubernetes.io/service-account.uid: 941a7698-5776-11e9-9f0e-b2722922aabd

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1716 bytes
namespace:  7 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...

Client side request:

Request URL: https://mydomain/apis/authorization.k8s.io/v1/selfsubjectaccessreviews
Request Method: POST
Status Code: 404
...
authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...

JS log:

log.js:11 10:06:23 AM Login Failed Error: Api request error: 
    at apiProxy.js:50
    at x (runtime.js:62)
    at Generator._invoke (runtime.js:288)
    at Generator.e.(/anonymous function) [as next] (https://dash.goto10.cz/static/js/2.00060ec4.chunk.js:1:583991)
    at r (asyncToGenerator.js:3)
    at a (asyncToGenerator.js:25)

Testing REST API call in Insomnia:

It works great with the same token.

I may have some free time this weekend I will try to make a fork. I think there's a bad endpoint for API calling used and since there's no exact logging what's called in request() function, logs didn't help much at this point.

[herbrandson]

Thanks for the info. I'm also hoping to have some time to investigate this weekend. Feel free to hit me up in the Kubernetes slack channel if you have any questions. My username is "Eric Herbrandson"

[frohikey]

I know why I'm not a big fan of proxies.
I fixed it by adding changeOrigin: true for proxy in the server.

Thus now it looks like this:

const proxySettings = {target, agent, secure: false, ws: true, changeOrigin: true, onError};

It works for me perfectly. Of course question is... how it runs in other environments.
When port-forwarded it still fails. Could you test it with this option?

tldr: But I finally logged in and found other bugs inside 😊

[herbrandson]

Thanks for tracking that down. I'll test with the changeOrigin: true setting.

Let me know what other bugs you find. I'll take a look asap.

[herbrandson]

@frohikey I've tested the changeOrigin: true setting and it doesn't seem to cause any issues for any of my clusters. I am really curious what could be different between our environments that makes this necessary for you but not for me (or others). Crazy.

Anyhow, thanks a TON for your help in tracking this down. When you've got a second, would you mind grabbing the latest herbrandson/k8dash:dev docker image and verifying that it resolves the issue for you?

Thanks

[adi90x]

Hello,
Just try the :dev tag and I got the same issue : Error occured attempting to login when using token to log ( token is working fine )
Logs are :

GET /static/js/main.74ddf54b.chunk.js 304
GET /oidc 304
GET / 200
[HPM] POST /apis/authorization.k8s.io/v1/selfsubjectaccessreviews -> https://10.43.0.1:443
POST /apis/authorization.k8s.io/v1/selfsubjectaccessreviews 201

Any idea of what I can test ?

[frohikey]

@herbrandson The latest dev image works great.

Well, I got 3 thumbs up with the issue so I was probably not alone 😁 I guess if it runs with ingress w/ controller there's one "extra hop" and original source is being lost. This option just forces using "x-http-headers" to keep it routed properly.

@adi90x It's probably another issue. You logs looks fine. It'd follow with redirect to root /. Post request to API returned 201 so the token entered was ok. Any errors in JS console in browser?

[adi90x]

@frohikey my bad I think my account service was not setup properly.

[herbrandson]

@frohikey Yeah, totally didn't to mean to imply that there wasn't a real issue there. Re-reading my last comment, it does come off as defensive. Sorry about that. That wasn't my intent. Thanks so much for your help tracking this down. I really appreciate the effort you put in!

@adi90x Sounds like you've got your issue resolved?

I think everything is resolved on this ticket at this point. I'll close it if I don't hear anything in the next day or two.

[frohikey]

@herbrandson No problem. That idea didn't even pop in my mind. And moreover I am no snowflake 😋 I know that hell trying to fix some issues when you can't reproduce it in your own env. Just basically the worst kind of hell. So I was happy to resolve it.

Keep up the good work!