websocket issues with basic auth within HTTPS about grafana-websocket-plugin

[bijwaard]

I noticed a lot of connections to the default live datasource in the grafana and apache log, could grafana be confused and try to use the default live datasource instead of my configured websocket datasource? Both the websocket and grafana are proxied on the same HTTPS URL <domain>:<port>, however the datasource wsVT-lab%20https has wss://<domain>:<port>/nl/enschede/lab/busbar/ws/ configured in the host field, while the live datasource has path /nl/enschede/lab/gf/api/live/ws on this URL, since the grafana root_url contains the path /nl/enschede/lab/gf.

[Tue Sep 13 14:31:17.372481 2022] [authz_core:debug] [pid 357717:tid 281471872913792] mod_authz_core.c(843): [client ::1:47382] AH01628: authorization result: granted (no directives), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.372815 2022] [authz_core:debug] [pid 357717:tid 281473231876480] mod_authz_core.c(843): [client ::1:47384] AH01628: authorization result: granted (no directives), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.372875 2022] [proxy:debug] [pid 357717:tid 281471872913792] mod_proxy.c(1503): [client ::1:47382] AH01143: Running scheme http handler (attempt 0), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.373053 2022] [proxy:debug] [pid 357717:tid 281473231876480] mod_proxy.c(1503): [client ::1:47384] AH01143: Running scheme http handler (attempt 0), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.373215 2022] [proxy:debug] [pid 357717:tid 281471872913792] proxy_util.c(2596): [client ::1:47382] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/annotations?from=1663079417335&to=1663079477336&limit=100&matchAny=false&dashboardUID=IK5EXKM4z to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.373313 2022] [proxy:debug] [pid 357717:tid 281471872913792] proxy_util.c(2819): [client ::1:47382] AH00947: connected /nl/enschede/lab/gf/api/annotations?from=1663079417335&to=1ny=false&dashboardUID=IK5EXKM4z to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20
[Tue Sep 13 14:31:17.373353 2022] [proxy:debug] [pid 357717:tid 281473231876480] proxy_util.c(2596): [client ::1:47384] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/ds/query to l://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.373643 2022] [proxy:debug] [pid 357717:tid 281473231876480] proxy_util.c(2819): [client ::1:47384] AH00947: connected /nl/enschede/lab/gf/api/ds/query to localhost:3000, refer <domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
::1 - - [13/Sep/2022:14:31:17 +0000] "GET /nl/enschede/lab/gf/api/annotations?from=1663079417335&to=1663079477336&limit=100&matchAny=false&dashboardUID=IK5EXKM4z HTTP/1.1" 200 319 "<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
::1 - - [13/Sep/2022:14:31:17 +0000] "POST /nl/enschede/lab/gf/api/ds/query HTTP/1.1" 200 486 "https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&vlab%20https&from=now-1m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
[Tue Sep 13 14:31:50.718382 2022] [proxy:debug] [pid 357717:tid 281473181520256] proxy_util.c(2596): [client ::1:47390] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/live/ws to lo
[Tue Sep 13 14:31:50.718458 2022] [proxy:debug] [pid 357717:tid 281473181520256] proxy_util.c(2819): [client ::1:47390] AH00947: connected /nl/enschede/lab/gf/api/live/ws to localhost:3000
::1 - - [13/Sep/2022:14:31:50 +0000] "GET /nl/enschede/lab/gf/api/live/ws HTTP/1.1" 400 338 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
[Tue Sep 13 14:32:50.903748 2022] [proxy:debug] [pid 357715:tid 281472678220160] proxy_util.c(2596): [client ::1:47404] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/live/ws to lo
[Tue Sep 13 14:32:50.903965 2022] [proxy:debug] [pid 357715:tid 281472678220160] proxy_util.c(2819): [client ::1:47404] AH00947: connected /nl/enschede/lab/gf/api/live/ws to localhost:3000

Kind regards,
Dennis

[bijwaard]

Hi Again,

Apparently, one of my problems was wrongly configured proxying of the grafana api/live/wx path in the apache proxy. This could be remedied with the following rule(s) in the secondary apache proxy (and similarly in the primary Internet-facing proxy):

        Define GrafanaPath /nl/enschede/lab/gf
        ProxyPassMatch    "${GrafanaPath}/api/live/(.*)$"      ws://localhost:3000${GrafanaPath}/api/live/$1
        ProxyPass          ${GrafanaPath}      http://localhost:3000${GrafanaPath}
        ProxyPassReverse   ${GrafanaPath}      http://localhost:3000${GrafanaPath}

After this fix, the live streaming was working with websockets local to grafana, such as ws://busbar/ws, but not with basic authentication. I was initially under the impression that the configured websocket was handled browser-side, but apparently it is handled server-side (using api/live/ws websocket within grafana localhost:3000 that re-streams all configured websockets to the browser). It is currently not possible to give basic-auth credentials with the websocket datasource configuration, so this won't work (yet) with server-side websockets.

It would be nice to also have browser-side support for websockets, such that the authentication would be handled by the browser and the authentication doesn't need to be configured&maintained in grafana.

Using brower-side websockets would also reduce the CPU load on the grafana server, e.g. on my embedded PC (nanoPI neo3), server-side websockets take +/- 10% of CPU core per 50Hz stream. With mulitple clients selecting different streams this may not scale. For example with four 50Hz streams:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                     
 472512 grafana   20   0  718956  20212  10816 S  24.4   2.0   1104:43 /var/lib/grafana/plugins/golioth-websocket-datasource/gpx_websocket_linux_arm64                                             
 415835 grafana   20   0 2057884  98876  33204 S  19.8   9.9   2292:06 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:de+

Kind regards,
Dennis

[chireia]

Hey @bijwaard , sorry about the lack of responses!
Great work, seems like you got it working, awesome.

When we first implemented the plugin, the only way to do authentication was by using the server-side handled by Grafana. Being transparent, I'm not aware that any changes on that behavior. Do you have any suggestions and/or experiences on doing that using only the client-side? Since Grafana introduced "Grafana Live" on their v8.0, plugin-wise development guides have being pretty basic and rudimentary. So we'd love to hear from you.

[bijwaard]

Hi @chireia,

Thanks for your message. I didn't pursue this much further, grafana seems to phase out browser-side support for other datasources as well (influxdb, prometheus, etc.). In my view browser-side support gives much more flexibility than forcing all data through the grafana server, and sometimes the grafana server may not be able to reach the datasource itself.

It is a pity that grafana (server-side) live streaming takes so much resources, and that it sometimes takes more than 10 seconds to start streaming when the dashboard is opened, and it has hick-ups when the dashboard refreshes other datasources. Our alternative is dashboard refresh at e.g. 1s interval, with our own JSON plugin back-end written in golang.

P.s. browser-side websockets with the same webserver-proxy works fine outside grafana, also with basic authentication.

Kind regards,
Dennis