After Publish connections are not returned to the pool about redis-py-cluster

[Grokzen]

The first iteration i did for the pubsub support i found out that the return from a publish command about how many clients that recieved the message was not accurate and it was only taking into account how many clients that was connected to that specific node that the publish command was talking to. Because of this, the compatibility with the existing publish/subscribe API and mechanics that is used by redis-py compatible software would be broken and redis-py-cluster could not act as a drop-in replacement to provide seamless compatibility. The only fix at that time was to have all clients talk to the same node because that would ensure that it would still work as expected.

Sinc then some things have changed because i now know that you can use the slot hashing mechanism to distribute the clients on all nodes but still do it in a predictable way and the pub/sub API would work as expected, however then you hit the performance problem that still plagues redis internally and that is described in the docs.

[Grokzen]

The pubsub implementation inside this lib is going to be removed and the work is happening in this PR #115. Pubsub should not be used in a cluster environment right now and you can read up on why here https://github.com/Grokzen/redis-py-cluster/blob/unstable/docs/Pubsub.md

The only time i will consider to add pubsub back into this lib, is redis itself reworks their implementation to be more cluster friendly.

[melhin]

Thanks for the update @Grokzen . But i just wanted to know why when i do a plain publish using the redis-cli the connections are returned and when i do a publish with the redis-cluster it hoggs on one particular node, any specific reason for that ?

[melhin]

Ok so i was still curious with the connection leakage problem with publish so after some digging around i found in rediscluster/connection.py get_connection there was this tag

• TODO: Pop existing connection if it exists.

So in release you were removing it from _in_use_pubsub_connections and then you blindly appended the connection to _available_pubsub_connections so a lot of publish connections stayed on and eventually causing the file limits on the system to get exhausted

I have created a pull request with this change . Let me know if im on the right track
#130

[Grokzen]

I worked some on this problem today and took a deeper look and compared my implementation to how redis-py handles connections in the pool and i saw several flaws with my implementation. I have now reworked the connection_pool code to be more similar to redis-py and how it works in there. It also means that it impacts pubsub code because those connections is no longer tracked in a different variables. You can see the code here #136 and it should superseed your PR #130 where it should fix the same problems and more.

[Grokzen]

Fixed in unstable and included in release 1.2.0