Does pipeline execute concurrently or sequentially ? and relationship between pipe and redis transaction? about redis-py-cluster

[Grokzen]

By default, pipelines executes sequentially. When a pipeline executes it will still use a pipeline object for each node in the cluster. If you want to run the pipeline in concurrent mode you can use the flag pipeline_use_threads=True to use a experimental feature to run each pipeline for each node with threads. From what i know it works really well.

Transactions is a hard problem to solve because there is several ways of defining a transaction and implementing support for both solutions is very hard because there is several corner cases that needs to be handled for the implementation to work. In short the 2 main options is to have a transaction pipeline across the entire cluster, or that the commands that is sent to each node should be transaction but they are not dependent on each other. So if one of 4 would fail the other 3 would still execute and be commited to redis. Currently i have documented and suggested that transactions is not supported right now and should not be used.

You should really read up on the docs, specially this one https://github.com/Grokzen/redis-py-cluster/blob/unstable/docs/Pipelines.md that describes the problems and my thoughts on the problem and possible solutions.

Regarding MGET in pipelines, it is currently disabled because because if you run MGET keyA keyB keyC keyD and 2 of the keys belongs to 1 node in the cluster and 2 keys belong to another node, i can't just split the command into MGET keyA keyC and MGET keyB keyD and send them off to their nodes and then handle any errors, moved, asking errors that pops up and retry until it either works or fails.
The only somewhat viable solution to this problem would be to take the MGET command and replace that command with a series of GET commands that each fetches the all keys and after execution is done, reassembles the original MGET to make it appear to work. In theory you can already do this yourself manually by having a helper function that you call when you want to run MGET that splits it into multiple GET and puts them on the pipeline stack, but it is not something that will be supported for now.

[Grokzen]

If that answers your questions, please close this issue. If you find some issue when trying something out, please open a new issue with a description of the problem. Also if you want to discuss this more, please join in the gitter chatroom (link in Readme) and ping me there.

[ppLorins]

hi @Grokzen #92 , this is my issue,by your answer , when I set pipeline_use_threads=True, all the commands queued on pipe will be sent to their coresponding node in the cluster respectively ? And pipe.executes() will returns only after all the responses arrived ? This is the way I supposed the pipe's behavior to be

my gitter quesion , just for record .