Error validating serviceaccount token when "pem_keys" are specified in the configuration about vault-plugin-auth-kubernetes

[jefferai]

What is the value of PUBLIC_KEY_FROM_OPENSSL_COMMAND?

[primeroz]

Hi Jeff.

After re-reading the case i wrote

This is the Key

sudo openssl ec -in /etc/kubernetes/ssl/serviceaccount.key -pubout
read EC key
writing EC key
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEzpOY+XvrouGrV84L5BiIVg8/x85xsav6
cbDHnTXcwiTP830Aeg7w18dS4RylgiWFx8+E/BnKD1VSdHCfWFsskAQXAcRMIPbN
scOKeyb+rupBpled89vbPNlKd0CUg+Vo
-----END PUBLIC KEY-----

I am actually configuring the kubernetes config through ansible ( https://github.com/TerryHowe/ansible-modules-hashivault/blob/master/ansible/modules/hashivault/hashivault_write.py ) and here is the output of the configured resource :

# vault read auth/kubernetes/config
Key                   Value
---                   -----
kubernetes_ca_cert    -----BEGIN CERTIFICATE-----
MIICJDCCAaqgAwIBAgIRAJHG+ckNRpez33xxJVHzK7MwCgYIKoZIzj0EAwMwUzEh
MB8GA1UEChMYbWludGVsLWs4cy1kZXYyLWJlM2JmZTA5MRYwFAYDVQQLEw1rdWJl
cm5ldGVzLWNhMRYwFAYDVQQDEw1rdWJlcm5ldGVzLWNhMB4XDTE4MDYxNDEyMjgz
M1oXDTI4MDYxMTEyMjgzM1owUzEhMB8GA1UEChMYbWludGVsLWs4cy1kZXYyLWJl
M2JmZTA5MRYwFAYDVQQLEw1rdWJlcm5ldGVzLWNhMRYwFAYDVQQDEw1rdWJlcm5l
dGVzLWNhMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEanE/Oj08T7MNdYRFrvYKGgMz
I0R9S7wyk6gLfNxaPCyL4Lnj9e1llzzbyufcKXM/CgqjVtjyzQAQIL8wVrbymRQ9
ayIB/V80m+qP0Eo/cBF3YHvCAQjOR+E/yvXtf47Wo0IwQDAOBgNVHQ8BAf8EBAMC
AqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUnLf6cQJ2jZ8tFtECzGefrQ6j
Mg4wCgYIKoZIzj0EAwMDaAAwZQIwIz79ZpySsHE8Bt5opMKxBvMVd3/TQuOQ9cCc
5VQIeAj2Iz6RTwpGtR1zZcmC1dP2AjEA+EO2j/FkeGGBkj1esaRMrLogMWCFoMAf
B9j+xO38SqyCpEKEnwtW9WS1MOtQwfsT
-----END CERTIFICATE-----
kubernetes_host       https://kubernetes
pem_keys              [-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEzpOY+XvrouGrV84L5BiIVg8/x85xsav6
cbDHnTXcwiTP830Aeg7w18dS4RylgiWFx8+E/BnKD1VSdHCfWFsskAQXAcRMIPbN
scOKeyb+rupBpled89vbPNlKd0CUg+Vo
-----END PUBLIC KEY-----]

The key does include the new lines.

I tried a few different combinations:

• \n after and before HEADER, single line key -----BEGIN PUBLIC KEY-----\nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEzpOY+XvrouGrV84L5BiIVg8/x85xsav6cbDHnTXcwiTP830Aeg7w18dS4RylgiWFx8+E/BnKD1VSdHCfWFsskAQXAcRMIPbNscOKeyb+rupBpled89vbPNlKd0CUg+Vo\n-----END PUBLIC KEY-----
  • Same error : {"errors":["asn1: structure error: length too large"]}
• No New line -----BEGIN PUBLIC KEY-----MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEzpOY+XvrouGrV84L5BiIVg8/x85xsav6cbDHnTXcwiTP830Aeg7w18dS4RylgiWFx8+E/BnKD1VSdHCfWFsskAQXAcRMIPbNscOKeyb+rupBpled89vbPNlKd0CUg+Vo-----END PUBLIC KEY-----
  • Error when writing: Exception: data does not contain any valid RSA or ECDSA public keys (Fair enough )
• \n instead of new lines
  • Error when writing: Exception: data does not contain any valid RSA or ECDSA public keys (Fair enough )

[primeroz]

So the only 2 way i manage to write the config to Vault is either by using New Lines for all lines or at least after and before the Headers.

All other cases i get an error because the string is not recognized as a valid public key

[jefferai]

I don't have a Kubernetes setup to test with, but I pulled out the function that parses the string and it works fine with your data:

package main

import (
        "crypto/ecdsa"
        "crypto/rsa"
        "crypto/x509"
        "encoding/pem"
        "errors"
        "log"
)

func main() {
        {
                val, err := parsePublicKeyPEM([]byte(pemBytes))
                if err != nil {
                        panic(err)
                }
                log.Printf("%#v\n", val)
        }
        {
                val, err := parsePublicKeyPEM([]byte(pemBytesNewline))
                if err != nil {
                        panic(err)
                }
                log.Printf("%#v\n", val)
        }
}

// PasrsePublicKeyPEM is used to parse RSA and ECDSA public keys from PEMs
func parsePublicKeyPEM(data []byte) (interface{}, error) {
        block, data := pem.Decode(data)
        if block != nil {
                var rawKey interface{}
                var err error
                if rawKey, err = x509.ParsePKIXPublicKey(block.Bytes); err != nil {
                        if cert, err := x509.ParseCertificate(block.Bytes); err == nil {
                                rawKey = cert.PublicKey
                        } else {
                                return nil, err
                        }
                }

                if rsaPublicKey, ok := rawKey.(*rsa.PublicKey); ok {
                        return rsaPublicKey, nil
                }
                if ecPublicKey, ok := rawKey.(*ecdsa.PublicKey); ok {
                        return ecPublicKey, nil
                }
        }

        return nil, errors.New("data does not contain any valid RSA or ECDSA public keys")
}

const (
        pemBytes = `-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEzpOY+XvrouGrV84L5BiIVg8/x85xsav6
cbDHnTXcwiTP830Aeg7w18dS4RylgiWFx8+E/BnKD1VSdHCfWFsskAQXAcRMIPbN
scOKeyb+rupBpled89vbPNlKd0CUg+Vo
-----END PUBLIC KEY-----`
        pemBytesNewline = "-----BEGIN PUBLIC KEY-----\nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEzpOY+XvrouGrV84L5BiIVg8/x85xsav6\ncbDHnTXcwiTP830Aeg7w18dS4RylgiWFx8+E/BnKD1VSdHCfWFsskAQXAcRMIPbN\nscOKeyb+rupBpled89vbPNlKd0CUg+Vo\n-----END PUBLIC KEY-----"
)

Can you try saving the value to a file (no special transformations), and using pem_keys=@file syntax in your vault write command?

[primeroz]

Hi Jeff, thanks for that !

I actually did the same thing short time ago and indeed the PublicKey is recognized and parsed properly by the

if ecPublicKey, ok := rawKey.(*ecdsa.PublicKey); ok {
                        return ecPublicKey, nil
                }

I think , with my extremely limited understanding of GO , is in the path_login.go , probably here where the Signature of the JWT token is checked against the public key.
This , to my understanding, happens before the call to the tokenreview kubernetes api , and i can see on my apiserver that no tokenreview api call happen when the error is shown.

I think that the error is happening before the tokenreview api and it must be when the signature and the jwt token are compared.

this would make sense since the write to the config succed and when no public_key is passed in the pem_keys argument that function exit just before the verify function

Just tried loading the files with the @file syntax and when trying to Login i get the same error as before

[jefferai]

Any chance you can add some logging into the code to try to track it down? For instance, is it coming from the JWS library via the parsedJWT.Validate call?

[primeroz]

I definetly can, but i am not sure i know how :)

I am currently running the official container v0.10.2 from docker hub, i never built vault with external plugin but i guess i can figure it out ( if you got any docs to point me to that would be helpful ) . My understanding is that i can just rebuild this auth plugin and load it into vault , right ?

If you could just provide me with the right syntax to add debug message i will add them into a bunch of places and will figure out where is bombing out

[jefferai]

The easiest thing might be to just build from Vault master (we've added a bit of debugging and pulled it in). You could then make a custom contianer that inherits from the 0.10.2 one but swaps in your binary.

[primeroz]

Will do an update, will be tomorrow though since is getting quite late here.

thanks

[primeroz]

Hi @jefferai

Error now confirm that the validation is failing with an error other than "Signature is invalid"
https://github.com/hashicorp/vault-plugin-auth-kubernetes/blob/master/path_login.go#L252

$ curl -X POST -d @payload.json https://vault:8200/v1/auth/kubernetes/login
{"errors":["failed to validate JWT: asn1: structure error: length too large"]}

I can also confirm that on https://jwt.io the Public key correctly verify the token.

If it helps you track down the issue i could send you a Signed token privately

[jefferai]

That would be great! Can you send to jeff at the company I work for dot com?

[primeroz]

@jefferai done

[jefferai]

@primeroz We figured it out -- the JWT library in use doesn't actually format (or validate) EC signatures according to spec. We're forking and patching it for now and longer-term will replace usage of the library. This should be fixed in 0.10.3.

[primeroz]

@jefferai Great, i will give it a try as soon as it is available .

I can't see any reference to this bugfix in the changelog for vault, is it the fix in anyway ?

Also i don't see an image for 0.10.3 on docker hub official repo, do you know when that should be available ?

[briankassouf]

@primeroz The docker image is now available on docker hub and this has been added to the CL.