SELECT queries are not delivered to the replica and shard to which token belongs to about gocqlx

[rkuska]

I believe that this should be reported against http://github.com/scylladb/gocql

[mmatczuk]

Can you provide IP/CIDR to DC mapping.

[rkuska]

Sent on slack.

[mmatczuk]

Interesting that all of the strange stuff goes from a single IP

michal@localhost:~/downloads$ for id in `cut -f1 -d\| selects_shard1_targets_different_from_coordinator.txt`; do
> grep $id id_to_client
> done
 913b6c10-23ee-11e9-a140-000000000025 |  10.127.251.3 
 9b315180-23ee-11e9-a140-000000000025 |  10.127.251.3 
 61ab5dc0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 7e7ed260-23ee-11e9-a140-000000000025 |  10.127.251.3 
 68097670-23ee-11e9-a140-000000000025 |  10.127.251.3 
 610be970-23ee-11e9-a140-000000000025 |  10.127.251.3 
 44f05320-23ee-11e9-a140-000000000025 |  10.127.251.3 
 2cc04ee0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 3f0806b0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 24c91fa0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 9c2611c0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 8ba3b0a0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 4b7a5dd0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6ba64c40-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6d33f120-23ee-11e9-a140-000000000025 |  10.127.251.3 
 90e0c8f0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 636d3200-23ee-11e9-a140-000000000025 |  10.127.251.3 
 4933c930-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6b5964c0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6cd1ace0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 8a915640-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6af63620-23ee-11e9-a140-000000000025 |  10.127.251.3 
 34cc1790-23ee-11e9-a140-000000000025 |  10.127.251.3 
 25681ec0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 20d2df30-23ee-11e9-a140-000000000025 |  10.127.251.3 
 7e3f5860-23ee-11e9-a140-000000000025 |  10.127.251.3 
 383cad40-23ee-11e9-a140-000000000025 |  10.127.251.3 
 3869fed0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 9c6d7b00-23ee-11e9-a140-000000000025 |  10.127.251.3 
 1eaacf10-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6d5f1fd0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 478a10d0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 67355d90-23ee-11e9-a140-000000000025 |  10.127.251.3 
 2c8e4260-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6981be90-23ee-11e9-a140-000000000025 |  10.127.251.3 
 89ab8a20-23ee-11e9-a140-000000000025 |  10.127.251.3 
 2b2f9cc0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6aed3570-23ee-11e9-a140-000000000025 |  10.127.251.3 
 7c4f9650-23ee-11e9-a140-000000000025 |  10.127.251.3 
 783ca620-23ee-11e9-a140-000000000025 |  10.127.251.3 
 2f186fb0-23ee-11e9-a140-000000000025 |  10.127.251.3 
 8c818d80-23ee-11e9-a140-000000000025 |  10.127.251.3 
 1dc0bd30-23ee-11e9-a140-000000000025 |  10.127.251.3 
 4877a520-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6da79a80-23ee-11e9-a140-000000000025 |  10.127.251.3 
 6d92da00-23ee-11e9-a140-000000000025 |  10.127.251.3 

While overall we have many of clients

 michal@localhost:~/downloads$ cut -f2 -d\| sessions.txt | sort | uniq
 10.127.251.11 
 10.127.251.12 
 10.127.251.13 
 10.127.251.15 
  10.127.251.3 
  10.127.251.5 
  10.127.251.6 
  10.127.251.7 
  10.127.251.8 
  10.127.251.9 
  54.38.84.185 
  54.38.84.186 
  54.38.84.187 
  54.38.84.188 
  54.38.84.189 
  54.38.84.190 
  54.38.84.191 
  54.38.84.192 
  54.38.84.193 
  54.38.84.241 
   54.38.84.97 
 78.46.232.116 

[mmatczuk]

Another strange thing is that this node made almost all the SELECT requests

 michal@localhost:~/downloads$ grep 10.127.251.3 sessions.txt | grep SELECT | wc -l
166
 michal@localhost:~/downloads$ grep SELECT sessions.txt | wc -l
189

[avikivity]

@vladzcloudius please give examples of failed queries and their traces (in a way that doesn't require combing through 10 attachments)

[mmatczuk]

My working theory is that 10.127.251.3 is not using the shard aware driver. I may be wrong, in that case it would mean that either there is a bug in gocql code that somewhat fails to get a token for a query and we are hitting [1] (looks like there is a room for improvement in that loop) or we are selecting wrong connection given a shard number (shard - connection mismatch in the driver). I created a dedicated branch https://github.com/scylladb/gocql/tree/mmt/juicy_debug that contains additional debug logging you need to build it with gocql_debug build tag.

[1] https://github.com/scylladb/gocql/blob/master/scylla.go#L120

[rkuska]

@mmatczuk I just checked and 10.127.251.3 is using the same image as other hosts so therefore it must be using the same shard aware driver.

[vladzcloudius]

@avikivity This github issue is not about queries that failed but rather about queries that were delivered to a coordinator and shard that are not owners of the replica for a corresponding token. And as a result one shard receiving a lot more requests than others as may be seen on the screenshot.

session_ids of such queries are a first field in the selects_shard1_targets_different_from_coordinator.txt file and can be retrieved as follows:

cat selects_shard1_targets_different_from_coordinator.txt | cut -d"|" -f1

All system_traces.sessions entries for such queries may be retrieved as follows (this would in particular include queries):

for sid in `cat selects_shard1_targets_different_from_coordinator.txt | cut -d"|" -f1`
do
    grep $sid sessions.txt
done

And the corresponding system_traces.events entries may be seen as follows:

for sid in `cat selects_shard1_targets_different_from_coordinator.txt | cut -d"|" -f1`
do
    grep $sid events.txt
done

[mmatczuk]

@vladzcloudius @rkuska I'm closing this issue here, we can move discussion to a different project or continue in upstream. I have opened issue gocql/gocql#1262 and submitted my fix gocql/gocql#1261. The ScyllaDB fork is already updated.

[vladzcloudius]

@haaawk Please, verify that our Java fork doesn't suffer from the same issues. The last time I checked - it does. ;)
Also, please, check with @mmatczuk - he found a few issues in the driver itself (not related to shard-awarness). There is a good chance that Java driver has same issues too.