Medusa not stopping cassandra as service properly. about cassandra-medusa

[adejanovski]

This can be overriden in the configuration file, where you can provide your own stop/start commands.
We usually don't care about properly draining the nodes because we're going to replace all the data anyway, which is why a "dirty" stop is fine, but I agree that the service should be stopped so you don't need to stop/start it again.
We're using /etc/init.d/cassandra start because we need to enforce the tokens in some cases, and pass them using a JVM flag. I guess we could workaround this by modifying the cassandra-env.sh file instead, although that could be a problem to some folks. We need to give this some thoughts.

It's interesting that /etc/init.d/cassandra stop returns before the node is actually down. Looks like we need a bit more checks to make sure it is down before proceeding with the cleanup and restore tasks.

[sandeepmallik]

@adejanovski Yes. Something like fetch PID of cassandra, stop cassandra and make sure PID doesn't exist.

root@ubuntu:# CassPid=$(pgrep -f cassandra)
root@ubuntu:# echo $CassPid
1376

[sandeepmallik]

@adejanovski After stopping cassandra, added sleep to fix it. This may not be right solution but works for me.

restore_node.py
logging.info('Waiting for cassandra to stop. Sleeping for 10 seconds')
time.sleep(10)

Also, verify with CQL is not working properly. It should be due to medusa is not waiting for CQL native transport port (9042) to be available before running select query. I added sleep(20) after cassandra starts then verify worked.

[2020-02-13 04:30:28,983] INFO: Executing restore verify query: select * from tutorialspoint.emp;
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/medusa/cassandra_utils.py", line 84, in new_session raise Exception('Could not establish CQL session after {attempts}'.format(attempts=attempts)) Exception: Could not establish CQL session after 5