Integrity check fails on passwords containing `$` [was: Auth server return bad url] about stunner

[vipcxj]

@rg0now

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: stunner-study-ai
  labels:
    {{- include "common.labels" . | nindent 4 }}
spec:
  controllerName: "stunner.l7mp.io/gateway-operator"
  parametersRef:
    group: "stunner.l7mp.io"
    kind: GatewayConfig
    name: stunner-study-ai
    namespace: stunner
  description: "STUNner is a WebRTC ingress gateway for Kubernetes"
---
apiVersion: v1
kind: Namespace
metadata:
  name: stunner
---
apiVersion: v1
kind: Secret
metadata:
  name: stunner-study-ai-auth
  namespace: stunner
type: Opaque
stringData:
  type: ephemeral
  secret: 4h$sdgh[k)tAgjTR54gfjus3wayrt
---
apiVersion: stunner.l7mp.io/v1
kind: GatewayConfig
metadata:
  name: stunner-study-ai
  namespace: stunner
spec:
  realm: stunner.l7mp.io
  authRef:
    kind: Secret
    name: stunner-study-ai-auth
    namespace: stunner
  # dataplane: default
  # authType: plaintext
  # userName: "user-1"
  # password: "pass-1"
  loadBalancerServiceAnnotations:
    kubernetes.io/elb.class: performance
    kubernetes.io/elb.id: XXXXXXXXXXX
    kubernetes.io/elb.lb-algorithm: LEAST_CONNECTIONS 
    kubernetes.io/elb.session-affinity-mode: SOURCE_IP
    kubernetes.io/elb.session-affinity-option: '{"persistence_timeout": "15"}'
    kubernetes.io/elb.health-check-flag: 'off'
    # kubernetes.io/elb.health-check-option: '{"delay": 3, "timeout": 15, "max_retries": 3}'
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: stunner-study-ai
  namespace: stunner
spec:
  gatewayClassName: stunner-study-ai
  listeners:
    - name: stunner-study-ai-udp-listener
      port: 3478
      protocol: TURN-UDP
---
apiVersion: stunner.l7mp.io/v1
kind: UDPRoute
metadata:
  name: stunner-study-ai
  namespace: stunner
spec:
  parentRefs:
    - name: stunner-study-ai
  rules:
    - backendRefs:
        - name: video-server
          namespace: default

Then my webrtc client achieve the auth config just from the stunner auth server, and unable to connect the peer.

04:04:58.347104 reconcile.go:180: stunner INFO: status: READY, realm: stunner.l7mp.io, authentication: static, listeners: NONE, active allocations: 0
04:04:58.347664 cds_client.go:83: cds-client INFO: connection successfully opened to config discovery server at ws://10.16.0.27:13478/api/v1/configs/stunner/stunner-study-ai?watch=true
04:04:58.348430 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
04:04:58.348492 server.go:28: stunner INFO: listener stunner/stunner-study-ai/stunner-study-ai-udp-listener: [turn-udp://10.16.0.140:3478<0:0>] (re)starting
04:04:58.348507 server.go:45: stunner INFO: setting up UDP listener socket pool at 0.0.0.0:3478 with 16 readloop threads
04:04:58.348865 server.go:166: stunner INFO: listener stunner/stunner-study-ai/stunner-study-ai-udp-listener: TURN server running
04:04:58.348874 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 2, changed objects: 1, deleted objects: 0, started objects: 1, restarted objects: 0
04:04:58.348883 reconcile.go:180: stunner INFO: status: READY, realm: stunner.l7mp.io, authentication: ephemeral, listeners: stunner/stunner-study-ai/stunner-study-ai-udp-listener: [turn-udp://10.16.0.140:3478<0:0>], active allocations: 0
04:05:21.743309 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
04:05:21.743356 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 2, deleted objects: 0, started objects: 0, restarted objects: 0
04:05:21.743366 reconcile.go:180: stunner INFO: status: READY, realm: stunner.l7mp.io, authentication: ephemeral, listeners: stunner/stunner-study-ai/stunner-study-ai-udp-listener: [turn-udp://10.16.0.140:3478<0:0>], active allocations: 0
04:06:41.100143 handlers.go:38: stunner-auth INFO: ephemeral auth request: username="1704690399:1" realm="stunner.l7mp.io" srcAddr=10.16.0.129:14411
04:06:41.100172 handlers.go:53: stunner-auth INFO: ephemeral auth request: success
04:06:41.100224 server.go:202: turn ERROR: Failed to handle datagram: failed to handle Allocate-request from 10.16.0.129:14411: integrity check failed
04:06:54.791511 handlers.go:38: stunner-auth INFO: ephemeral auth request: username="1704690412:1" realm="stunner.l7mp.io" srcAddr=10.16.0.129:2687
04:06:54.791550 handlers.go:53: stunner-auth INFO: ephemeral auth request: success
04:06:54.791618 server.go:202: turn ERROR: Failed to handle datagram: failed to handle Allocate-request from 10.16.0.129:2687: integrity check failed
04:07:11.944288 handlers.go:38: stunner-auth INFO: ephemeral auth request: username="1704690430:1" realm="stunner.l7mp.io" srcAddr=10.16.0.129:19056
04:07:11.944317 handlers.go:53: stunner-auth INFO: ephemeral auth request: success
04:07:11.944381 server.go:202: turn ERROR: Failed to handle datagram: failed to handle Allocate-request from 10.16.0.129:19056: integrity check failed
04:07:16.036893 handlers.go:38: stunner-auth INFO: ephemeral auth request: username="1704690434:1" realm="stunner.l7mp.io" srcAddr=10.16.0.129:19341
04:07:16.036921 handlers.go:53: stunner-auth INFO: ephemeral auth request: success

After I change authType to plaintext, the issue disappears. Of couse, change the secret to a shorter one also works.

[rg0now]

An immediate problem is that your UDPRoute cannot attach to the Gateway: parentRef.Name must match the name of the parent Gateway, not the name of the name of the listener:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
  name: owt-media-plane
  namespace: stunner
spec:
  parentRefs:
    - name: owt-udp-gateway
  rules:
    - backendRefs:
        - name: owt-server
          namespace: default

You can see why this in a problem the stunnerd logs:

08:54:03.331956 reconcile.go:157: stunner WARNING: running with no clusters: all traffic will be dropped

If you want to attach the UDPRoute to a specific listener on the gateway, you must specify both the Gateway name and the listener name in the parentRef:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
  name: owt-media-plane
  namespace: stunner
spec:
  parentRefs:
    - name: owt-udp-gateway
      sectionName: owt-udp-listener
  rules:
    - backendRefs:
        - name: owt-server
          namespace: default

This though still doesn't explain the other weirdness, the TURN errors:

08:54:03.421273 server.go:194: turn ERROR: error when handling datagram: failed to create stun message from packet: unexpected EOF: not enough bytes to read header

This means the client sends truncated packets or some middlebox chops off the meaningful part of our TURN packets or this can be the wildest case of an MTU issue I've ever seen. I must see a tcpdump to know what's wrong here, but it is definitely not a STUNner issue.

[rg0now]

Well, and then there is the auth issue. This is weird: we fail to generate a turn protocol scheme and a transport protocol, that is clearly a bug. Did you install from the stable channel or the latest version from the dev channel? There has been a massive rewrite lately that might have broken the dev installs, we'll look into that ASAP.

[vipcxj]

@rg0now Thank you for your quick response. I found I can't connect the turnserver, it return 401, so I change the auth type from ephemeral to static, The problem remains. Then I found your response, And change the config. It works. Then I change the auth type back, I can't get to the server again. In the log, I found:

10:11:39.029140 handlers.go:38: stunner-auth INFO: longterm auth request: username="1695381093:12345" realm="stunner.l7mp.io" srcAddr=192.168.0.82:38130
10:11:39.029170 handlers.go:53: stunner-auth INFO: longterm auth request: success
10:11:39.029240 server.go:194: turn ERROR: error when handling datagram: failed to handle Allocate-request from 192.168.0.82:38130: integrity check failed

What is it means?

Well, and then there is the auth issue. This is weird: we fail to generate a turn protocol scheme and a transport protocol, that is clearly a bug. Did you install from the stable channel or the latest version from the dev channel? There has been a massive rewrite lately that might have broken the dev installs, we'll look into that ASAP.

I install the stable operotor weeks ago. At first it return good url. Today I found the url is broken. I have not reinstalled the operator.

I change the auth type back to static, and I can connect to server again, Here is the log:

10:20:50.456079 server.go:194: turn ERROR: error when handling datagram: failed to create stun message from packet: unexpected EOF: not enough bytes to read header
10:20:52.997995 handlers.go:25: stunner-auth INFO: plaintext auth request: username="user-1" realm="stunner.l7mp.io" srcAddr=192.168.0.82:13830
10:20:53.050236 server.go:194: turn ERROR: error when handling datagram: failed to create stun message from packet: unexpected EOF: not enough bytes to read header
10:20:53.089937 server.go:194: turn ERROR: error when handling datagram: failed to create stun message from packet: unexpected EOF: not enough bytes to read header
10:20:53.121255 server.go:194: turn ERROR: error when handling datagram: failed to create stun message from packet: unexpected EOF: not enough bytes to read header
10:20:53.214464 allocation.go:290: turn INFO: No Permission or Channel exists for 10.233.75.18:39127 on allocation 10.233.97.159:32869
10:20:53.256763 handlers.go:25: stunner-auth INFO: plaintext auth request: username="user-1" realm="stunner.l7mp.io" srcAddr=192.168.0.82:13830
10:20:53.256809 handlers.go:84: stunner-auth INFO: permission granted on listener "stunner/owt-udp-gateway/owt-udp-listener" for client "192.168.0.82:13830" to peer 10.233.75.18 via cluster "stunner/owt-media-plane"
10:20:55.920946 server.go:194: turn ERROR: error when handling datagram: failed to create stun message from packet: unexpected EOF: not enough bytes to read header
10:20:55.975054 server.go:194: turn ERROR: error when handling datagram: failed to create stun message from packet: unexpected EOF: not enough bytes to read header

By the way, there are logs of similar errors such as "not enough bytes to read header" and "BadFormat for message/cookie: 34353637 is invalid magic cookie (should be 2112a442)". It seems that the stunner still work well with these errors.

[rg0now]

This was me breaking the auth service, which, combined with anyone bug in our helm charts that defaults the auth-service container image version to the latest dev version even in the stable install, caused that everyone was immediately affected.

We'll try to settle this ASAP. Meanwhile you can manually downgrade the auth-service container image version to v0.15.0, that should fix this until we stabilize the dev version and roll a new release. Or you can use a fix TURN credential for now.

As per the truncated TURN messages, that's still unexplainable. I doubt this issue has anything to do with the auth service bug, maybe a misbehaving client? We'll see once we fix the auth-service.

[vipcxj]

Here is my stunner image:

docker.io/l7mp/stunnerd:0.15.0
docker.io/l7mp/stunner-auth-server:dev

I have change docker.io/l7mp/stunner-auth-server:dev to docker.io/l7mp/stunner-auth-server:0.15.0

The url is right now. But ephemeral auth type still not work.

10:58:15.365317 handlers.go:38: stunner-auth INFO: longterm auth request: username="1695383890:12345" realm="stunner.l7mp.io" srcAddr=192.168.0.82:6508
10:58:15.365345 handlers.go:53: stunner-auth INFO: longterm auth request: success
10:58:15.365429 server.go:194: turn ERROR: error when handling datagram: failed to handle Allocate-request from 192.168.0.82:6508: integrity check failed

And the error not change.

[rg0now]

From the below it seems that the auth-credentials are accepted by STUNner.:

10:58:15.365345 handlers.go:53: stunner-auth INFO: longterm auth request: success

The subsequent error is then caused by that your client generates a wrong integrity hash into the packets:

10:58:15.365429 server.go:194: turn ERROR: error when handling datagram: failed to handle Allocate-request from 192.168.0.82:6508: integrity check failed

Can you please confirm that it works with static username/password pairs, only the ephemeral auth credentials are what trigger this issue? Judging from the weird errors you get, it can be some misbehaving TURN client. Are you trying to connect from a browser?

[rg0now]

I cannot reproduce the ephemeral auth issue with the latest dev version. Please uninstall stunnerd and the gateway-operator and reinstall from the dev channel and report back any issue you find. Thx!

As per the "truncated TURN packets problem: please use the simple-tunnel tutorial to check whether your cluster is OK. That demo uses our own TURN client so at least the "misbehaving client" problem will go away.

[vipcxj]

From the below it seems that the auth-credentials are accepted by STUNner.:

10:58:15.365345 handlers.go:53: stunner-auth INFO: longterm auth request: success

The subsequent error is then caused by that your client generates a wrong integrity hash into the packets:

10:58:15.365429 server.go:194: turn ERROR: error when handling datagram: failed to handle Allocate-request from 192.168.0.82:6508: integrity check failed

Can you please confirm that it works with static username/password pairs, only the ephemeral auth credentials are what trigger this issue? Judging from the weird errors you get, it can be some misbehaving TURN client. Are you trying to connect from a browser?

I have tried this many times and only ephemeral auth triggers this issue. I can't say for sure if this has been an issue before, as this seems to be the first time I've tried to use ephemeral auth. it's always been static auth before, I'll do some further research next Monday.

[vipcxj]

Ephemeral auth still not work. Here are the errors:

04:04:58.347104 reconcile.go:180: stunner INFO: status: READY, realm: stunner.l7mp.io, authentication: static, listeners: NONE, active allocations: 0
04:04:58.347664 cds_client.go:83: cds-client INFO: connection successfully opened to config discovery server at ws://10.16.0.27:13478/api/v1/configs/stunner/stunner-study-ai?watch=true
04:04:58.348430 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
04:04:58.348492 server.go:28: stunner INFO: listener stunner/stunner-study-ai/stunner-study-ai-udp-listener: [turn-udp://10.16.0.140:3478<0:0>] (re)starting
04:04:58.348507 server.go:45: stunner INFO: setting up UDP listener socket pool at 0.0.0.0:3478 with 16 readloop threads
04:04:58.348865 server.go:166: stunner INFO: listener stunner/stunner-study-ai/stunner-study-ai-udp-listener: TURN server running
04:04:58.348874 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 2, changed objects: 1, deleted objects: 0, started objects: 1, restarted objects: 0
04:04:58.348883 reconcile.go:180: stunner INFO: status: READY, realm: stunner.l7mp.io, authentication: ephemeral, listeners: stunner/stunner-study-ai/stunner-study-ai-udp-listener: [turn-udp://10.16.0.140:3478<0:0>], active allocations: 0
04:05:21.743309 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
04:05:21.743356 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 2, deleted objects: 0, started objects: 0, restarted objects: 0
04:05:21.743366 reconcile.go:180: stunner INFO: status: READY, realm: stunner.l7mp.io, authentication: ephemeral, listeners: stunner/stunner-study-ai/stunner-study-ai-udp-listener: [turn-udp://10.16.0.140:3478<0:0>], active allocations: 0
04:06:41.100143 handlers.go:38: stunner-auth INFO: ephemeral auth request: username="1704690399:1" realm="stunner.l7mp.io" srcAddr=10.16.0.129:14411
04:06:41.100172 handlers.go:53: stunner-auth INFO: ephemeral auth request: success
04:06:41.100224 server.go:202: turn ERROR: Failed to handle datagram: failed to handle Allocate-request from 10.16.0.129:14411: integrity check failed
04:06:54.791511 handlers.go:38: stunner-auth INFO: ephemeral auth request: username="1704690412:1" realm="stunner.l7mp.io" srcAddr=10.16.0.129:2687
04:06:54.791550 handlers.go:53: stunner-auth INFO: ephemeral auth request: success
04:06:54.791618 server.go:202: turn ERROR: Failed to handle datagram: failed to handle Allocate-request from 10.16.0.129:2687: integrity check failed
04:07:11.944288 handlers.go:38: stunner-auth INFO: ephemeral auth request: username="1704690430:1" realm="stunner.l7mp.io" srcAddr=10.16.0.129:19056
04:07:11.944317 handlers.go:53: stunner-auth INFO: ephemeral auth request: success
04:07:11.944381 server.go:202: turn ERROR: Failed to handle datagram: failed to handle Allocate-request from 10.16.0.129:19056: integrity check failed
04:07:16.036893 handlers.go:38: stunner-auth INFO: ephemeral auth request: username="1704690434:1" realm="stunner.l7mp.io" srcAddr=10.16.0.129:19341
04:07:16.036921 handlers.go:53: stunner-auth INFO: ephemeral auth request: success

Change to static auth, error disappeared, and it works.

[rg0now]

I cannot reproduce this issue with latest stable. I tested with simple-tunnel, changed the authentication to ephemeral:

kubectl apply -f - <<EOF
apiVersion: stunner.l7mp.io/v1
kind: GatewayConfig
metadata:
  name: stunner-gatewayconfig
  namespace: stunner
spec:
  authRef:
    kind: Secret
    name: stunner-auth-secret
    namespace: stunner
  dataplane: default
  realm: stunner.l7mp.io
---
apiVersion: v1
kind: Secret
metadata:
  name: stunner-auth-secret
  namespace: stunner
type: Opaque
stringData:
  type: ephemeral
  secret: my-shared-secret
EOF

Then tested with turncat:

cd <stunner>
export IPERF_ADDR=$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}")
go run cmd/turncat/main.go --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener  udp://$IPERF_ADDR:5001
iperf -c localhost -p 5000 -u -i 1 -l 100 -b 8000 -t 5
...
[  1] 0.0000-5.0114 sec  5.18 KBytes  8.46 Kbits/sec   3.593 ms 0/53 (0%) 16.024/12.296/101.769/8.420 ms 10 pps 10/0(0) pkts 0.065999

[vipcxj]

@rg0now I try again, still not work, same error. Is there a docker image bundled with turncat, so I can use it to test?

[levaitamas]

Hi @vipcxj !

@rg0now I try again, still not work, same error. Is there a docker image bundled with turncat, so I can use it to test?

l7mp/net-debug has turncat built-in.

[vipcxj]

@levaitamas I found the problem~ It's secret key.
I use the secret key in your example, it works , then change back to our secret key, it faild.
Here is our secret key:

apiVersion: v1
kind: Secret
metadata:
  name: stunner-study-ai-auth
  namespace: stunner
type: Opaque
stringData:
  type: ephemeral
  secret: 4h$sdgh[k)tAgjTR54gfjus3wayrt
  # secret: my-shared-secret

[rg0now]

This is (hopefully) resolved now. Feel free to reopen if problem persists.

[vipcxj]

@rg0now which commit solve it? I have shorten the secret as the workaround. So it seems that I will not trigger this issue any more.

[rg0now]

Sorry for the confusion, I thought we resolved it here. Can you please confirm that the secret 4h$sdgh[k)tAgjTR54gfjus3wayrt does not work while my-shared-secret does? If yes, they I'll re-label this as a bug and try to look into it.

[vipcxj]

Can't replace secret right now, but I confirm that I did have this issue when I commented earlier, and it's not clear if you've done any fixes since then. I fixed it by use a shorter secret. So it seems that too long secret will tirgger the issue.

[rg0now]

Thx for the report, I'm trying to track this down now. My bet would be that it's the symbols like $ and the like that make the operator go south, but we'll see.

[rg0now]

So this seems like a good old UNIX/Bash gotcha.

If you use the below to set the secret, then any value you try to use that is containing a symbol like $ in stringData gets corrupted:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: stunner-auth-secret
  namespace: stunner
type: Opaque
stringData:
  type: ephemeral
  secret: "4h$sdgh[k)tAgjTR54gfjus3wayrt"
EOF
kubectl -n stunner get secrets stunner-auth-secret --template={{.data.secret}} | base64 -d
4h[k)tAgjTR54gfjus3wayrt

The good news is that the corrupted secret nicely finds its way into the STUNner dataplane config, so at least we don't mess things up during the base64 encode/decode roundtrip:

stunnerctl -n stunner config udp-gateway -o jsonpath='{.auth.credentials.secret}'
4h[k)tAgjTR54gfjus3wayrt

However, place your Secret into a YAML file, say, /tmp/my-secret.yaml, like this:

cat /tmp/my-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: stunner-auth-secret
  namespace: stunner
type: Opaque
stringData:
  type: ephemeral
  secret: "4h$sdgh[k)tAgjTR54gfjus3wayrt"

And then kubectl-apply it and all is fine:

kubectl apply -f /tmp/my-secret.yaml
kubectl -n stunner get secrets stunner-auth-secret --template={{.data.secret}} | base64 -d
4h$sdgh[k)tAgjTR54gfjus3wayrt
stunnerctl -n stunner config udp-gateway -o jsonpath='{.auth.credentials.secret}' 
4h$sdgh[k)tAgjTR54gfjus3wayrt

So I guess shell-escaping bites us here: the shell interprets $sdg in the secret as a shell variable and subtitutes it with its value (empty string). Let's try to escape the $ in the secret (observe how we now use \$ now):

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: stunner-auth-secret
  namespace: stunner
type: Opaque
stringData:
  type: ephemeral
  secret: "4h\$sdgh[k)tAgjTR54gfjus3wayrt"
EOF
kubectl -n stunner get secrets stunner-auth-secret --template={{.data.secret}} | base64 -d
4h$sdgh[k)tAgjTR54gfjus3wayrt

So this fixes it, and this is indeed a shell escaping thingie...

Today I (re)learned something important again: just use standard YAML manifests that do not get processed by the shell and then all is fine.

[vipcxj]

@rg0now I got the auth secret from stunner auth server, so if the secret is changed by something, it not matter what I gotten. Because I do not achieve the secret from k8s directly.

[vipcxj]

@rg0now And I have tried config secret both directly in GatewayConfig and an authRef of secret, nothing changed. The issue does not disappeared until I changed the secret. By the way, the new secret does not contain '$', but still contain '@', and it is 13 bytes long.

[rg0now]

Can you please specify exactly what didn't work for you? Just post all YAMLs and command lines what you did so that I can reproduce the problem.

[rg0now]

This should be fixed in 1bb46b7 and will be available in dev once the CI pipeline has finished building the image.

The bug was in STUNner: we apply environment substitution on the config file while parsing it (this allows to customize per-gateway configs to per-pod context locally) and this plays badly with TURN credential parsing if the password or the secret contains a $ symbol. In such cases we think everything that comes after the $ is the name an environment variable and we try to substitute it (to the empty string).

Thanks a lot @vipcxj for helping us tracking this down, this was a particularly ugly one. Feel free to reopen if the problem persists.