Unexpected response code: 500 (rpc error: No path to datacenter): Sync not working, Services not being de-registered about marathon-consul

[dankraw]

Hi @tomwganem

Unexpected response code: 500 (rpc error: No path to datacenter)

This looks like a Consul problem, the agent (used by marathon-consul to communicate with Consul) is not able to send any rpc messages to Consul servers, as it has not yet joined the cluster correctly. Please check the agents health by calling their http api, whether you are able to fetch services from the Consul catalog etc., or via Consul UI console.

[tomwganem]

It was a consul problem. consul ports were inadvertently closed.

However, there is still an issue with running in federated consul. We run federated consul to stich together two mesos cluster installations running in an active-passive configuration. We we run marathon-consul in our passive site against a consul cluster that is configured to not be in "dc1", it de-registers all services in our active site ("dc1") whenever it does a sync.

[jessp01]

In my particular case (consul used with Prometheus), the datacenter field in /etc/consul.d/consul.json contained upper-cased letters, lower-casing everything fixed the issue. Thought I'd comment on that in case anyone else encounters the same issue.

[tomwganem]

Should I close this ticket and create a new ticket for this federated consul issue?

[tomwganem]

closing. New issue in #286

[dankraw]

@tomwganem thanks for explanation :)