Ping Timeouts? about kazoo

[bbangert]

The main reason I didn't account for this was because I assumed that writing to the dead connection would throw a socket error and cause the loop to exit and reconnect. How is the socket closed, yet can be written to for the ping?

[justinrosenthal]

Great question. To be honest, I don't know the answer but that's exactly what's happening. The ZooKeeper logs very clearly state that the connection has been closed, the ephemeral data for that node is gone, but the client doesn't think anything out of the ordinary has happened: no state change, etc.

It's possible that virtualization might have something to do with it (I've seen the issue on EC2). Oddly enough, each time it's happened, the socket does close after 15 minutes -- and seemingly exactly 15 minutes every time.

However, it seems like a ping timeout may be useful in other situations, such as: if the ZooKeeper server is misbehaving (pegged CPU, swapping, etc?). Getting the client onto a healthy node before it's session expires would be great.

[bbangert]

Agreed, if a ping is not returned within (how many seconds exactly?), then the client should close its connection and try to reconnect.

Are you using the 0.9 release, or the latest beta?

In the latest beta, I think the best way to handle this would be for it to set an event when it sends a ping, and clear it when it gets a ping reply. It can check this event to see if a ping was already sent should it time-out in the loop again, and if it hasn't gotten the prior ping raise a ConnectionClosed exception.

[justinrosenthal]

We've been using 1.0b1 and will be submitting a pull request shortly that implements the timeout as you suggested.

The timings: submit ping at 1/3 the timeout interval, expect pong within another 1/3 the timeout interval. If the pong isn't received, you then have 1/3 the timeout interval to connect to another server. This seemed to be the suggested approach on zookeeper-users.

[bbangert]

Let me know if this failed to resolve the issue and I'll reopen. Thanks!

[cablehead]

Cheers Ben, we appreciate it. Will do..