`partisan_plumtree_broadcast` crashing about partisan

[aramallo]

@mcesaro I need to take a look at this to see whether the issue is in Partisan, PlumDB or Erleans (our fork).

[aramallo]

@mcesaro Thanks for the extra info. However, notice that since plum_db is in your logs, you are not running just Partisan, or you running plum_db directly? Otherwise, you are still running erleans (plum_db is a dependency in our fork).

[mcesaro]

@aramallo Hi,
it's happening also on non-erlans applications:

=CRASH REPORT==== 26-Oct-2023::08:28:08.676084 ===
  crasher:
    initial call: partisan_plumtree_broadcast:init/1
    pid: <0.18966.0>
    registered_name: partisan_plumtree_broadcast
    exception error: undefined function plum_db:merge/2
      in function  partisan_plumtree_broadcast:handle_cast/2 (/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl, line 549)
      in call from gen_server:try_handle_cast/3 (gen_server.erl, line 1103)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1165)
    ancestors: [partisan_sup,<0.1007.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1008.0>,<0.1019.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [125007732914024471|257206006587947701]}}]
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 28
    reductions: 40368
  neighbours:

=SUPERVISOR REPORT==== 26-Oct-2023::08:28:08.676618 ===
    supervisor: {local,partisan_sup}
    errorContext: child_terminated
    reason: {undef,
                [{plum_db,merge,
                     [{{{erleans_pm,registry},
                        {'pebble@max-a5',
                            {<<"01HDNAMM2S470TFV43NFD2GMNE">>,pebble}}},
                       [{{12,'e8a18d8a-73c7-11ee-bbf0-1d718ad5fce5@max-a5'},
                         2}]},
                      {object,
                          {[{{12,
                              'e8a18d8a-73c7-11ee-bbf0-1d718ad5fce5@max-a5'},
                             2,
                             [{'$deleted',{1698,301688,672946}}]}],
                           []}}],
                     []},
                 {partisan_plumtree_broadcast,handle_cast,2,
                     [{file,
                          "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
                      {line,549}]},
                 {gen_server,try_handle_cast,3,
                     [{file,"gen_server.erl"},{line,1103}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1165}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.18966.0>},
               {id,partisan_plumtree_broadcast},
               {mfargs,{partisan_plumtree_broadcast,start_link,[]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,5000},
               {child_type,worker}]

I think I can trigger this behaviour in my laptop by suspending it on Wifi1 and waking it on Wifi2.
It seems that when there is a network change, partisan still try to use the last known network.
Admittely not a common use case, but still it might be a nuisance. A potential workaround would be to have a function/API to signal the network change and reset all the connections; otherwise, the effect is that the running nodes are stuck in the try-to-reconnect loop, but also that new nodes cannot join the cluster.

[mcesaro]

@aramallo Hi, my bad. One of the servers also includes erleans as a dependency. So, I re-checked the dependencies for all the applications; I'm not explictly using erleans in a distributed mode as only one server is currently running erleans, but the network swtich issue now is limited to the host running erleans with plum_db.