I have a setup with libcluster and three cluster nodes, and I sometimes get errors like the following.
18:59:50.813 [warn] [swarm on app@server] [tracker:handle_topology_change] handoff failed for {CommandedApplication, Commanded.Event.Handler, "Projectors.Object"}: {{%RuntimeError{message: "attempted to call GenServer #PID<0.3862.0> but no handle_call/3 clause was provided"}, [{Commanded.Event.Handler, :handle_call, 3, [file: 'lib/gen_server.ex', line: 773]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 661]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 690]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}, {GenServer, :call, [#PID<0.3862.0>, {:swarm, :begin_handoff}, 5000]}}
18:59:50.813 [error] GenServer #PID<0.3487.0> terminating
** (RuntimeError) attempted to call GenServer #PID<0.3862.0> but no handle_call/3 clause was provided
(commanded 1.2.0) lib/gen_server.ex:773: Commanded.Event.Handler.handle_call/3
(stdlib 3.12.1) gen_server.erl:661: :gen_server.try_handle_call/4
(stdlib 3.12.1) gen_server.erl:690: :gen_server.handle_msg/6
(stdlib 3.12.1) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:DOWN, #Reference<0.1551621646.1983119361.7440>, :process, #PID<0.3862.0>, {%RuntimeError{message: "attempted to call GenServer #PID<0.3862.0> but no handle_call/3 clause was provided"}, [{Commanded.Event.Handler, :handle_call, 3, [file: 'lib/gen_server.ex', line: 773]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 661]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 690]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}
I'm not sure how to debug this. This means that the projectors stop working. When I restart a node they start again but then I get the same kinds of errors and not all projectors are running.
I don't really care if there are errors in the log, but I want my projectors to eventually be running, even if it takes a few restarts when taking a node down or up.
My setup looks like this: in my application I start the projectors as children to my application supervisor. Is there a good way to make sure that my projections are not going away when restarting a node? Should I look for something in particular in the logs?