Connection Pool Bug about redis-py-cluster

[Grokzen]

@JoelBender Hi. First thing is that i would not recommend to use pubsub with a redis cluster. There is currently alot of problems with the implementation inside redis iteself that makes it very slow and has crazy ammount of overhead. In short, it is implemented in a way that for each published message it will be sent to all nodes in the cluster because it is implemented so that any client can connect to any node and listen to any channel. This means that the fanout has to be implemented in the way it is. The only sane suggestion i have yet seen is to use the same hashslot mechanics that is already implemented for all other commands where the slot is determined by the key. By doing this in the pubsub case, it would remove the need to distribute the published messages to all nodes and then it would be really easy for clients to implement this pattern.

Anyhow, i still find it interesting that the client behaves this way. It should be releasing back the connection after a published message O.o. I will do some digging during the week and see what i can find out.

[JoelBender]

Thank you! I completely understand why it shouldn't be used as it stands now, and I will be using MQTT for my very simple pubsub application requirements. I intend to also dig into this bug at some point, but it will be quite some time before I get the chance.

[Grokzen]

This issue is similar to #129 and and the provided solution could fix this issue also.

[Grokzen]

@JoelBender I worked some on this problem today and took a deeper look and compared my implementation to how redis-py handles connections in the pool and i saw several flaws with my implementation. I have now reworked the connection_pool code to be more similar to redis-py and how it works in there. It also means that it impacts pubsub code because those connections is no longer tracked in a different variables. You can see the code here #136 where it should fix this problem and more.

[Grokzen]

@JoelBender Connection pool logic works much better in 1.2.0 and when i test things out by publishing alot of messages with a limited number of connections i do not see this problem with reaching the max number of connections because they are now added back to the pool properly when the command is done. You will however still get to the max number of connections used if you would to run with say max connections = 16 and use the same client object in say 32+ threads and max out the number of publish commands sent. But this is intended behavior and you would see the very same thing in redis-py.

If yo find some other or similar errors in release 1.2.0, please open a new ticket.