Orchestrator reliance on backend mysql server about orchestrator

[shlomi-noach]

For instance, if the backend mysql is unavailable can Orchestrator still do failovers based on some cached state? Can it failover it's own backend (hence dogfood)?

It cannot. You have three options available:

• running a circular master-master setup, with your own failover/routing mechanism. We do that. Generally master-master is undesired and to be avoided. orchestrator guarantees no conflict of writes on a master-master setup and is safe to use. We use haproxy checks to route orchestrator mysql traffic to the active/available master.
• running Galera/xtradb cluster/InnoDB cluster
• wait; we're working on raft consensus for orchestrator, where nodes would be independent of each other yet collaborating, each with their own backend store. This would be either mysql or sqlite.

[sjmudd]

I have an orchestrator backend with slaves and have tested "scheduled" failover to a slave. This is usually to achieve MySQL upgrades or change the backend due to hardware issues etc. This type of failover works fine though it's not quite what you're asking about. Do not expect orchestrator to be able recover from its own backend failure.

However, if you can arrange, on orchestrator backend failure, for the following to be true orchestrator will recover and that is probably all you care about:

• all connections to the old master are killed
• the new "master" is writeable (obviously)
• the name that orchestrator uses to talk to its backend (vip name) does not change but points to the new host
• the data that was in the old host is present in the new one

Under these conditions orchestrator will detect the backend failure but will recover from it to no ill effect.

The main reason that orchestrator is unable to recover itself is that currently a lot of state it may use for recovery is stored in the database with no "caching layer" running on the active node. That is not available at the moment the orchestrator backend fails.

I personally, for performance reasons, would like to see the state being managed by the active node but that requires a very significant amount of code change to achieve that. Given the complexity of all the situations that orchestrator can handle such a change if it were ever to happen needs to be made with a considerable amount of care.

[bbeaudreault]

Thanks guys