LEAK: ByteBuf.release() was not called before it's garbage-collected. about reactor-netty

[violetagg]

@subham611 Please follow this FAQ and provide the extended logs
https://projectreactor.io/docs/netty/release/reference/index.html#faq.memory-leaks

[violetagg]

@subham611 Consider checking Spring Boot supported versions
https://spring.io/projects/spring-boot#support

[github-actions]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[subham611]

I updated spring boot to latest version 3.2.4 still getting this error.
I have added following jvm parameters, and not seeing any additional log

-XX:MaxDirectMemorySize=1024M
-Dio.netty.leakDetection.targetRecords=40
-Dio.netty.leakDetection.level=paranoid
-Dlogging.level._reactor.netty.channel.LeakDetection=debug

Do I need to set logging.level._reactor.netty.channel.LeakDetection=debug in application.properties.

[violetagg]

@subham611 Try to set it in application.properties

[subham611]

yeah, saw this #1874 (comment)
Setting following properties in application.properties file

spring.netty.leak-detection=paranoid
logging.level._reactor.netty.channel.LeakDetection=DEBUG

[subham611]

@violetagg I set above mentioned spring properties but not seeing any change in log message

{"timestamp":1713180562148,"thread":"reactor-http-epoll-1","logger":"io.netty.util.ResourceLeakDetector","message":"LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
	org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:72)
	org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39)
	org.springframework.core.codec.CharSequenceEncoder.encodeValue(CharSequenceEncoder.java:90)
	org.springframework.core.codec.CharSequenceEncoder.lambda$encode$0(CharSequenceEncoder.java:74)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
	reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103)
	reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2909)
	reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241)
	reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
	reactor.core.publisher.Mono.subscribe(Mono.java:4568)
	reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:202)
	reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:63)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
	reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:297)
	reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:478)
	reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:470)
	reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	reactor.core.publisher.MonoZip$ZipCoordinator.request(MonoZip.java:220)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	reactor.core.publisher.MonoZip.subscribe(MonoZip.java:129)
	reactor.core.publisher.Mono.subscribe(Mono.java:4568)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234)
	reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
	reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onError(MonoIgnoreElements.java:84)
	reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onError(FluxConcatArray.java:208)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	reactor.netty.FutureMono$FutureSubscription.operationComplete(FutureMono.java:193)
	io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
	io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
	io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
	io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
	io.netty.util.concurrent.PromiseCombiner.tryPromise(PromiseCombiner.java:170)
	io.netty.util.concurrent.PromiseCombiner.access$600(PromiseCombiner.java:35)
	io.netty.util.concurrent.PromiseCombiner$1.operationComplete0(PromiseCombiner.java:62)
	io.netty.util.concurrent.PromiseCombiner$1.operationComplete(PromiseCombiner.java:44)
	io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
	io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
	io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
	io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
	io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754)
	io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339)
	io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:691)
	io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:735)
	io.netty.channel.AbstractChannel$AbstractUnsafe.handleWriteError(AbstractChannel.java:950)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:933)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:570)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

[violetagg]

@subham611 It doesn't seem that the changes are applied.

[violetagg]

You should be able to see output similar to the one here #1874 (comment)

[subham611]

Is there any other way to set netty property?

[violetagg]

@subham611 Please consult with the reference documentation of the technologies that you use. For Spring Boot - it should be application.properties (there might be different versions of this - for testing, production etc), then the environment where you run your application - whether the provided JVM OPTS are applied etc.

I cannot help without knowing the environment where you run your app.

You should be able to check the exact JVM OPTS that are used to run your app and the binaries that are published.

[subham611]

I have added this flag -Dlogging.level.io.netty.util.ResourceLeakDetector=debug as well and i do see following logs

{"timestamp":1713191024585,"thread":"main","logger":"io.netty.util.ResourceLeakDetector","message":"-Dio.netty.leakDetection.level: paranoid","context":"default","log_level":"DEBUG","is_logging_enabled":false}
{"timestamp":1713191024585,"thread":"main","logger":"io.netty.util.ResourceLeakDetector","message":"-Dio.netty.leakDetection.targetRecords: 40","context":"default","log_level":"DEBUG","is_logging_enabled":false}

Which confirms that leakDetection level is set to paranoid but somehow I am not seeing full logs during error.

I have setup a rate limiter using webFilter and I am suspecting some issue there.

@Component
public class RateLimiterFilter implements WebFilter {
  private final RateLimiter rateLimiter;
  @Override
    public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
       if(!rateLimiter.acquirePermission()) {
           log.warn("Too many requests");
           exchange.getResponse().setStatusCode(HttpStatus.TOO_MANY_REQUESTS);
           return Mono.empty(); // I am suspecting this might be causing some issue. 
       }
       return chain.filter(exchange);
    }
}

Let me remove this and try. ref: #518 (comment)