Why is `publish` blocked in pipeline mode? about redis-py-cluster

[Grokzen]

PubSub is something that i do not reccomend using when running a native redis cluster, it has its problems with scalability and probably need some more work on the redis side before it is mature enough to be used reliably.

I think that in the start i did block the creation of a pubsub class that would work inside a cluster, but i opened it up after some work with the code and you can now use pubsub with the guarantees that the API in from redis-py will behave the same way in this lib. The current implementation however is made so that all pubsub calls go to only 1, but the same node every time and this preservs the API. The other way to do pubsub in a cluster is to take the channel as a key and hash it to determine what node PUBLISH & SUBSCRIBE should talk to. This however breaks the API and i could not guarantee libs would be compatible and behave as expected.

Another major blocker for pubsub in a cluster is the performance problem with the current implementation. There is a thread on google groups that you can read up on some of the problems https://groups.google.com/forum/?hl=sv#!topic/redis-db/B0_fvfDWLGM

Because of all this i decided that pubsub would not be part of pipelines because they are already very complex to deal with and opening pubsub up is probably not something i will do any time soon because you already have it available through the main client class. If you want pubsub i would reccomend that you do not run it on a redis cluster and you should run it on a non cluster server.

[leporid]

Thanks. My use case does not involve very much load on the pubsub system, so I'm not worried about performance currently. Furthermore, I am passing pipeline_use_threads=False to the StrictRedisCluster initializer, so the pipeline doesn't do much more than group the results of some serial commands, in my understanding. With this in mind, I simply unblocked publish for pipelines, in the interest of being able to use the same client code as before migrating to redis-py-cluster.

[Grokzen]

If you find that you can use that implementation then go ahead :] But i would still suggest you do not use cluster pubsub.

If this question is answered then please close this issue. If you find some other specific problem with pubsub when using it or you have some enhancement ideas, please open a new issue ticket and i can take a look at it.

[leporid]

Sure, thanks again!