Library doesn't handle non contiguous key set about redis-py-cluster

[Grokzen]

@inspired1 I do not clearly understand where/what the problem is that you have. Could do a more explicit write-up on what redis server version you are using, what code you are trying to run and perhaps a full stack-trace of the problem you encounter? Otherwise it is hard for me to help you :]

And i do not really know what non contiguous key sets block answers are so could you please explain abit more about that?

[inspired1]

First of all:
thank you for the fast answer.
I'm using redis 3.0.0.beta6.
and latest redis-py-cluster library

Library ask for CLUSTER NODES to each node and redis node answers the following line:
4d9b013b9fa2a33bd63ffeceba3e1c96c0a0a8f :0 myself,master - 0 0 1 connected 0-5460 [5477-<-ae2b08435b1f7073a2bd6edee7708d46b4296593] [5485-<-ae2b08435b1f7073a2bd6edee7708d46b4296593] [5493-<-ae2b08435b1f7073a2bd6edee7708d46b4296593] [5497-<-ae2b08435b1f7073a2bd6edee7708d46b4296593] [5540-<-ae2b08435b1f7073a2bd6edee7708d46b4296593] [5544-<-ae2b08435b1f7073a2bd6edee7708d46b4296593] [5554-<-ae2b08435b1f7073a2bd6edee7708d46b4296593]....

you setted a variable range_ which contains u'[5477-<-ae2b08435b1f7073a2bd6edee7708d46b4296593]'
so, when in line 121 you do:
first, last = range_.split("-")
an exception is thrown because the string contains 2 "-"

and anyway this type of answer by a redis node is not supported by libray

[inspired1]

I suggest somethig like:

                    for range_ in slots:
                        try:
                            if "-" in range_:
                                first, last = range_.split("-")
                            else:
                                first = last = range_
                            for i in range(int(first), int(last) + 1):
                                self.slots[i] = addr
                        except ValueError as e:
                            k,a = range_[1:-1].split('-<-')
                            self.slots[int(k)] = self.resolve_name(a)

followed by a method who resolve names
i.e. following :

    def resolve_name(self,name):
        if not hasattr(self,'_cached_names'):
            self._cached_names = {}
        import re
        if name not in self._cached_names:
            for node in self.startup_nodes:
                try:
                    r = self.get_redis_link(node["host"], node["port"])
                    self._cached_names[filter(lambda x : 'myself' in x[2], map(re.compile('\s').split,r.execute_command('cluster nodes').splitlines()))[0][0]] = node
                except:
                    pass
        return self._cached_names[name]

[Grokzen]

What cluster setup do you have? how many masters and how many slaves for each master? because i cant replicate that output from CLUSTER NODES command that you get when i run that command on 3 masters with each one slave.

I currently get the following output from CLUSTER NODES when i run on the latest commit of the 3.0 redis branch.

127.0.0.1:7000> cluster nodes
22997cee73affec2ff1e00f9567db51e46ae8494 127.0.0.1:7002 master - 0 1403872604652 3 connected 10923-16383
e8491ca741aad4f555074fcdf348506135f9870d 127.0.0.1:7001 master - 0 1403872605653 2 connected 5461-10922
c6dbe465a396813c72f5d3442e5006e487d11ef0 127.0.0.1:7005 slave 22997cee73affec2ff1e00f9567db51e46ae8494 0 1403872604652 6 connected
d9a8818d9022a740afd7865459b439e03a7685e5 127.0.0.1:7004 slave e8491ca741aad4f555074fcdf348506135f9870d 0 1403872605653 5 connected
243a6578d676e002a48d9d8f1a6d7ee93c96952c 127.0.0.1:7000 myself,master - 0 0 1 connected 0-5460
081ea0a5ecaae1e8a460eded92050bd652369bc0 127.0.0.1:7003 slave 243a6578d676e002a48d9d8f1a6d7ee93c96952c 0 1403872603649 4 connected

When i can replicate your setup on my side i can merge your code :]

Is it possible that you can try the latest commit of 3.0 branch and see if there is any difference in your output from CLUSTER NODES ??

[inspired1]

Hi Grok,
I'm using 3 machine with 3 master and 3 slaves
this is output redis-trib.rb check

M: efed0cc56671fb4c992993f3f0d1d21ec3f7edc6 192.168.1.127:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: ee3525d93948125c72ceb520088c03c3d4cb787a 192.168.1.253:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 61fd11644a5801bd3c05eb2e2a91ba62098c1c25 192.168.1.127:7004
   slots: (0 slots) slave
   replicates ee3525d93948125c72ceb520088c03c3d4cb787a
S: 64422d6e6fee694c2c32709132a805dd1bf61db9 192.168.1.253:7005
   slots: (0 slots) slave
   replicates efed0cc56671fb4c992993f3f0d1d21ec3f7edc6
S: a3a001639325602cd3b46f75d43c5080bb64bcac 192.168.1.156:7003
   slots: (0 slots) slave
   replicates a1373a7d5ab015b656b4216201aaa92d565f2f78
M: a1373a7d5ab015b656b4216201aaa92d565f2f78 192.168.1.156:7002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

It works fine if cluster has no configuration change but, when a node dies, it is relaunched by a process monitor I did, in some cases redis cluster attach new node just created to the cluster and (I don't know how, the new node is configured to serve a non contiguous key group. In this case cluster nodes answers something like previous post. I think this behaviour means that not all slots are covered by cluster node.
The very strange behaveur is that while You ask for cluster node to another node in the cluster ti answers well (as You wrote in previous post).
so I can imagine cluster python library can query another node if a node answers in that strange way.

Tank You
inspired1

[Grokzen]

Thanks @inspired1 i think i can test it on my end now :] I will get back when/if i get it to fail.

[mattsta]

Also note: the next Redis Cluster release will have support for CLUSTER SLOTS so you don't have to manually parse a bulk text output: redis/redis@e14829d

Proper documentation will be added soon, but it's already in the Redis unstable branch if you want to try it yourself.

[Grokzen]

@mattsta Ye i have seen that new command and implementing that is in on my TODO list :] But how would CLUSTER SLOTS command behave when the cluster is in the state that is described in this post? Would it work better than CLUSTER NODES and i do not have to care about this wierd case?

[mattsta]

The original output format at the top is new to me. After looking at the Redis cluster source, that output happens when slots are migrating: see https://github.com/antirez/redis/blob/unstable/src/cluster.c#L3391-L3404 — so, someone added a new instance, then a subset of the existing slots migrate to the new instance.

The CLUSTER SLOTS output will only return the live master followed by usable (non-dead) replicas. I don't think the migrating status reported in CLUSTER NODES represents anything usable by clients; it's just informational for performance or networking troubleshooting.

As always, the only way to have 100% cluster access is to follow MOVED failures. The "Use slot to node mapping" way of sending commands is a (very very useful) optimization, but at any time the node map can be out of date and we have to be ready to handle MOVED failures at any time too.

[Grokzen]

@inspired1 I will start to rework the code to use CLUSTER SLOTS and then it should fix your issue.

Thanks for the quick help @mattsta :]

[inspired1]

thank you

[Grokzen]

@inspired1 Please see referenced commit and try if it works for you or not. It works alot better on my side atleast. Remember that from now you need a redis server that supports CLUSTER SLOTS and i do not think that it exists in beta.6 so you need beta.7 for your testing.

[Grokzen]

@inspired1 I am going to close this issue because the new code works very well for me and if you find that it fails in the same case as it did before then please reopen this issue and ping me again :]

[inspired1]

Ok, thank You @Grokzen I didn't do a deep test, and it seems works fine. Thank You again.