RabbitMQ UnexpectedConnectionDriverException leads to the deadlock about logback-gelf

[pukkaone]

What version of the com.rabbitmq:amqp-client dependency are you running? Version 4.0.0 introduced a change to ForgivingExceptionHandler.java to log the error. In previous versions, it did nothing with the error. As a work around, can you downgrade the amqp-client version?

[AleAndForCode]

I'm running on amqp-client:5.12.0. As a temporary workaround, I replaced the GelfAMQPSender class with the Channel#waitForConfirms(long timeout) call instead of the Channel#waitForConfirms() one.

...
                BasicProperties properties = propertiesBuilder.build();

                channel.basicPublish(
                        exchangeName,
                        routingKey,
                        properties,
                        toAMQPBuffer(message.toJson()).array());
                channel.waitForConfirms(waitForConfirmsTimeout); // trying to prevent deadlock when the AMQP Connection thread logs the MissedHeartbeatException

                return true;
...

If it's correct solution, I can make PR with the waitForConfirmsTimeout config parameter for the GelfAppender.

[pukkaone]

I'm surprised your workaround mitigates the issue. It's a fundamental design flaw. The logger invokes amqp-client to send messages. The amqp-client invokes the logger to log errors. This circular dependency leads to the deadlock.

Another possible solution is configure logback to not send log events from amqp-client to the GelfAppender.

[AleAndForCode]

I suppose that deadlock can occur when the app tries to log any message via amqp connection with an unresponsive peer. The waitForConfirms() or the waitForConfirms(0) methods call Object#wait() and thread may wait forever (except spurious wakeup). If log events from amqp-client aren't send to the GelfAppender and amqp connection became disrupted, I'm not sure that any next GelfAppender sending will not lead to the same issue.

The amqp-client invokes the logger to log MissedHeartbeatException exceptions from another tread.

If the waitForConfirmsTimeout expires, a TimeoutException will be thrown. That will recreate the channel, connection and increments tries. Finally, the connection will be recovered or we will get the ErrorStatus.