Live queries fail and cause the query to be run over and over on the host about fleet

[zwass]

Thank you for the details on the LB. Ultimately I think you're going to have to invest the time into sorting that out to get the full live query functionality.

I'm going to continue investigating the "unhappy state" issue and see if we can get that sorted for you.

[zwass]

I believe I have resolved the cleanup issue in #2316. I'd like to close out this issue with the merge of that PR. Please feel free to follow up with a new issue if we need to make additional changes to support sticky connection with your LB.

[zwass]

Fleet is definitely supposed to clean up queries that are in weird execution states, so that's likely a bug that I will have to look into.

Are you able to connect the browser directly to one of the Fleet servers rather than the load balancer? It would be great to see if queries work over standard websocket, or at least rule out that the issue is to do with the SockJS XHR transport.

[crimsonknave]

I was able to connect directly (or, at least directly to the k8s service url) and websockets work perfectly. However, I need to be on the load balancer url for our SSO/SAML integration to work. I had to use a service account that doesn't have SAML to connect.

I am also able to get queries working via fleetctl. I can certainly work with that and don't need live queries working in the UI, but it would be great to get a fix for the isssues that arise when a live query is run in the ui when websockets don't work.

I guess for the the issues are:

1. Staging works via xhr and prod does not
2. Production gets stuck in a bad state when a live query is run via the 'official' url

I'm very confused by the fact that in prod the first request to xhr_send gets a 200 and the second gets a 404. Unless that error code is being thrown for a very abnormal reason the url should be present, it was just successful a moment ago.

[zwass]

I am going to address the clean up issue for queries that end up in a bad state.

Can you provide more details about your LB so that I can try to reproduce the XHR issue locally?

[zwass]

I think the XHR issue between staging and production is the multiple Fleet servers behind the load balancer.

Are you able to configure the LB to make the connections sticky? With Websockets the connection is persistent, so it doesn't much matter what the LB does. For XHR to work, the consecutive requests have to hit the same backend server.

Our XHR fallback (via SockJS) is a relatively simple workaround and would require substantial modifications to support load balancing with non-sticky connections.

[zwass]

If you need it to achieve sticky sessions, we can configure Fleet/SockJS to set a JSESSIONID cookie (or any other name necessary).

[crimsonknave]

@zwass I don't think I'm going to have time to debug the LB issues anytime soon. I'm satisfied with running the queries via fleetctl using a service account dedicated to it (pointing to the kubernetes service url). If I never get the live queries to work in the UI that's fine for right now, but it'll be great when the cleanup ships and we don't have to worry about someone who doesn't know/remember get Fleet in an unhappy state.

The LB stuff is going to be a bit complicated. It's an nginx proxy that integrates with Okta to check if you are allowed to access the url. So, it's a load balancer as well as IAM tool. There was a setting that was supposed to enable websockets that we tried, but didn't have any luck. In addition, that load balancer points to the kubernetes service url for all the clusters, so there's another layer to add complexity to.