Management DB crash if two connections have the same name about rabbitmq-management

[simonmacmullen]

It looks like the management DB ended up storing details for multiple different connection processes under the same name. We assume that each connection has a unique name.

So either there's some way for connection names to clash, or they are not getting removed correctly in this case.

[martink2]

Maybe i am able to contribute a little to this, we also have the stomp plugin running and due to:

rabbitmq/rabbitmq-stomp#7

we quite frequently have stale connections with the same src_ip:src_port -> dst_ip:dst_port as running
connections, when we try to close them via the management frontend nothing happens.

Once we have those entries, the node running the management db keeps allocating memory until
it crashes. I recently managed to attach an observer with the following result, which showed all
memory going into the process with the name rabbit_mgmt_db.

I can provide the dump if needed.

Thanks

Martin

[michaelklishin]

Interesting. rabbit_mgmt_db is part of the management plugin.

Do MCollective clients immediately re-connect after being disconnected?

[michaelklishin]

Also, do you configure the management plugin in any way?

[martink2]

As to the reconnect: yes they do.

As to the config:

{rabbitmq_management, [
    {listener, [{port, 15671},
                {ssl, true}
              ]},
    {sample_retention_policies,
          [{global,   [{60, 5}, {3600, 60}, {86400, 1200}]},
           {basic,    []},
           {detailed, []}]
    }
  ]}

[martink2]

Yesterday i updated rabbitmq to the latest version and now i get a new message from the web ui

The management statistics database currently has a queue of 385536 events to process. If this number keeps increasing, so will the memory used by the management plugin. You may find it useful to set the rates_mode config item to none.

With the queue constantly growing and with it the memory of cause.

On the Cluster i can see the following duplicate connections:

10.173.0.251:33639 -> 10.67.76.151:61613
10.173.0.251:33639 -> 10.67.76.151:61613
10.173.0.251:33639 -> 10.67.76.151:61613
10.173.1.58:39900 -> 10.67.76.151:61613
10.173.1.58:39900 -> 10.67.76.151:61613

With rabbitmqctl report showing :

<[email protected]>   10.173.0.251:33639 -> 10.67.76.151:61613 (1)    <[email protected]>   1   guest   /mcollective    false   false   10  0   0   0   0   0   0   running
<[email protected]>   10.173.0.251:33639 -> 10.67.76.151:61613 (1)    <[email protected]>    1   guest   /mcollective    false   false   11  0   0   0   0   0   0   running
<[email protected]>   10.173.0.251:33639 -> 10.67.76.151:61613 (1)    <[email protected]>   1   guest   /mcollective    false   false   10  0   0   0   0   0   0   running

trying to kill the connection vie rabbitmqctl results in:

rabbitmqctl close_connection '<[email protected]>' "go away"
Closing connection "<[email protected]>" ...
Error: {not_a_connection_pid,<5604.28426.120>}

Erlang observer cant find the pid either, is there any way to manually delete the entry from the
table ?

Thanks

Martin

[michaelklishin]

See rabbitmq/rabbitmq-stomp#8. We identify connections using STOMP socket's IPs and ports. So at some point ephemeral port range rolls over and we get duplicate connection names.

I believe this can be closed. Please leave a comment if you think otherwise.

[SeriousSem]

Sorry for writing in the closed thread, but I'm just curious whether this issue has already fixed in the version 3.5.2, because we can still see channels with the same name in the management plugin and still get error like I posted in the rabbitmq-users forum(link in the first message of this thread).

Thanks in advance
Sem

[michaelklishin]

@SeriousSem rabbitmq/rabbitmq-stomp#8 has made it into 3.5.2 and the fix was confirmed by @martink2.

[SeriousSem]

@michaelklishin we are still facing this problem in the 3.5.2.

=ERROR REPORT==== 10-Jun-2015::14:19:14 ===
webmachine error: path="/api/channels/{ip}"
{exit,
    {{{case_clause,[[<0.7527.210>],[<0.14129.210>]]},
      [{rabbit_mgmt_db,created_event,3,[]},
       {rabbit_mgmt_db,handle_call,3,[]},
       {gen_server2,handle_msg,2,[]},
       {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,247}]}]},
     {gen_server2,call,
         [{global,rabbit_mgmt_db},
          {get_channel,<<"{ip} -> 0.0.0.0:0 (1)">>,
              {no_range,
                  {range,1433938690000,1433938750000,5000},
                  no_range,no_range}},
          infinity]}},
    [{gen_server2,call,3,[]},
     {rabbit_misc,with_exit_handler,2,[]},
     {rabbit_mgmt_wm_channel,is_authorized,2,[]},
     {webmachine_resource,resource_call,3,[]},
     {webmachine_resource,do,3,[]},
     {webmachine_decision_core,resource_call,1,[]},
     {webmachine_decision_core,decision,1,[]},
     {webmachine_decision_core,handle_request,2,[]}]}

Any points what could I forget to eliminate this error?

[michaelklishin]

@SeriousSem as rabbitmq/rabbitmq-stomp#8 explains, the issue is in the event emitting part. Do you use STOMP? Something else? Do you observe connections that never go away?

A single stack trace does not explain much. I highly recommend starting a discussion on the mailing list first.