Failover logic appears to be broken about hector

[rantav]

Which version are you using?
I think this is a bug fixed at the latest version.
It was fixed in this commit http://github.com/rantav/hector/commit/1c664c55b3561e64177c193ded6c3f80f377c6d9
and the version you need is 0.6.0-9 or -10 or higher
(or, if you work with cassandra 0.5.1 then 0.5.1-9)

[aleph2012]

I just got the latest code from your master branch. I checked, and it does have the changes for the commit you mention. But it still exhibits the same behavior. Seems like one way to fix it would be to have a try catch around the call to client = clientPools.borrowClient() in skipToNextHost(). Then after catching the exception it could try the next host in the list.

[rantav]

Hi sorry for delay, I'm on vacation but am looking into it... just a little slow...

[aleph2012]

No problem! Enjoy your vacation and forget this junk! :) I have a workaround in my version for now.

[rantav]

do you have two hosts in your cassandra setup?

[rantav]

nm my question about two hosts, it's irrelevant; I see the problem, will fix it soon

[rantav]

Fixed
$ git push --all
Counting objects: 116, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (62/62), done.
Writing objects: 100% (85/85), 9.41 KiB, done.
Total 85 (delta 40), reused 0 (delta 0)
To [email protected]:rantav/hector.git
1962aa7..1a80908 0.5.0 -> 0.5.0
80babe5..43c35f5 0.5.1 -> 0.5.1
079633b..e614402 0.6.0 -> 0.6.0
5cf91dc..4321125 master -> master

Aleph, can you please validate? Thanks!

[aleph2012]

Hey rantav, the new logic seems to work! I had a follow-up question though. I think we only query cassandra once for its set of hosts in initFailover(). So if in the meantime we add more hosts dynamically, and later on we get a failure I think the failover logic will only loop thru the hosts it knew about at startup.

[rantav]

yes, we only query once that's true.
however, there's a jmx call updateKnownHosts which should be triggered by an admin once new hosts are added.
I've considered polling for this but I think the admin-way is a less simpler and maybe it's all that's needed.

[aleph2012]

I didn't know about the JMX call. Yeah, I agree, the admin should trigger it when adding new hosts. Not a big fan of polling in general. I'll make sure our admin knows about this! :)