Dataplane not getting information from the ControlPlane in Hybrid mode about kong

[lays147]

@ADD-SP

Do you have some deployment that might share/change the kong_prefix directory across different data_plane instances?

No, there's no configuration changes between deployments.

And do you have steps to help us to reproduce this issue locally?

So, the scenario is like I told on the bug report. I have one control plane, and two data planes in my production environment running on ECS. The autoscaling configuration(based on cpu and memory consumption) is very sensitive, so it scales out very drastically(3-to-5).

And when this happens, some or all of the new containers have one or more errors like the ones that I added, and does not load the configuration from the control plane. The containers appear to be in a deadlock.

I don't know if this can be reproduced locally. I might throw a hand, and maybe you can try using some Kubernetes stuff like k6, adding very little resources to the dataplane node, and forcing it to scale in the cluster and see if these issues happen. The problem happens on trying to add new data plane nodes to an existent environment.

One thing that might stabilize my services, is changing the health check from the /status to /status/ready route, what would avoid the data planes without control plane communication to serve traffic. I'm going to deploy this change today, and hope that it stabilizes everything.

[ADD-SP]

@lays147 Thanks for your report. Seems the file of LMDB is corrupted. Now Kong is using LMDB to store entities on the data_plane. Do you have some deployment that might share/change the kong_prefix directory across different data_plane instances?

[ADD-SP]

And do you have steps to help us to reproduce this issue locally?

[StarlightIbuki]

The first 2 errors looks like a network glitch and if it does not persists it should be fine to ignore them;
Could you check and share the status of LMDB cache file?

[lays147]

@StarlightIbuki I don't have access to the underlying infrastructure to check that file.

And apparently, after changing the health check, it fixed this issue for me on the events of autoscaling.

[StarlightIbuki]

@StarlightIbuki I don't have access to the underlying infrastructure to check that file.

And apparently, after changing the health check, it fixed this issue for me on the events of autoscaling.

Good to know it works for you.