IllegalMonitorStateException in ThreadPoolExecutor with Redisson Integration about blockhound

[pderop]

thanks for reporting, will take a look.

[pderop]

I could reproduce using this sample project:

403-repro.tgz

now, it seems that there are some BlockingOperationError exception that are happening but they are not logged.
For example, I have hacked the BlockHound BlockingOperationError class like this:

    public BlockingOperationError(BlockingMethod method) {
        super(String.format("Blocking call! %s", method));
        this.method = method;
        StringWriter stringWriter = new StringWriter();
        PrintWriter printWriter = new PrintWriter(stringWriter);
        this.printStackTrace(printWriter);
        System.out.println("[" + Thread.currentThread().getName() + "] BlockingOperationError: " + stringWriter.toString());
    }

then I can see a tons of exceptions like this:

[redisson-3-1] BlockingOperationError: reactor.blockhound.BlockingOperationError: Blocking call! jdk.internal.misc.Unsafe#park
	at reactor.blockhound.BlockHound$Builder.lambda$new$0(BlockHound.java:260)
	at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:490)
	at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89)
	at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
	at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
	at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
	at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
	at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

there is a permitBlockingCalls method in FastThreadLocalThread which can be used to mark a FastThreadLocalThread so it can run blocking tasks, but I don't know what is exactly happening for the moment.

while trying to debug, I observed for example that RedisClient can schedule some FastThreadLocalRunnable tasks like in the following stacktrace:

java.lang.Exception: Stack trace
	at java.base/java.lang.Thread.dumpStack(Thread.java:2209)
	at io.netty.util.concurrent.FastThreadLocalRunnable.<init>(FastThreadLocalRunnable.java:24)
	at io.netty.util.concurrent.FastThreadLocalRunnable.wrap(FastThreadLocalRunnable.java:38)
	at io.netty.util.concurrent.DefaultThreadFactory.newThread(DefaultThreadFactory.java:105)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
	at org.redisson.client.RedisClient$1$1.run(RedisClient.java:288)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:31)
	at java.base/java.lang.Thread.run(Thread.java:1583)

so, it looks like some blocking codes are scheduled in some netty FastThreadLocalThreads, which are assumed to be non blocking.

honestly, I'm not sure to understand everything here, but I cannot see any issues in BlockHound itself for the moment.

[cuongqluu]

I think I can skip blocking inside by
BlockHound.builder().allowBlockingCallsInside(Vertx.class.getName(), "executeBlocking").install(),
but it seems to skip all blocking operations. Can you guide me how I can find the exactly class and methods to use allowBlockingCallsInside().

[pderop]

I tried it like the following, but it does not seem to resolve the problem:

		BlockHound.install(builder -> builder.allowBlockingCallsInside("io.vertx.core.Vertx", "executeBlocking"));

However, I tried this which works, but it's probably not a good idea to allow blocking calls from netty FastThreadLocalThread, maybe the following could be done with adding a flag in order to only allow blocking calls during the startup time ? The following allows all blocking calls from netty FastThreadLocalThread threads:

		BlockHound.install(builder -> builder.nonBlockingThreadPredicate(p ->
				thread -> thread instanceof FastThreadLocalThread ? false : p.test(thread)));

you can also do this, which allow blocking calls from any FastThreadLocalRunnable.run methods (but is it what should be done, I'm not sure):

		BlockHound.install(builder -> builder.allowBlockingCallsInside("io.netty.util.concurrent.FastThreadLocalRunnable", "run"));

you can check (Dis-)allowing blocking calls inside methodssection from docs for more examples.

[draguljce]

I've hit the same issue and managed to work around it like so:

BlockHound.install(builder -> {
            builder.nonBlockingThreadPredicate(p -> thread -> !thread.getName().contains("redisson") && p.test(thread));
        });

[cuongqluu]

It seems to the Redisson does not support Reactive completely, so maybe allow Redisson call blocking inside.
@pderop the builder.nonBlockingThreadPredicate(..) works for me.
But the builder.allowBlockingCallsInside(..) does not work.
@draguljce I tried your code and it just passed the redisson startup time. Because Redisson still call FastThreadLocalRunnable in runtime.

[pderop]

if you don't mind, I will close this issue, because for the moment I don't see any issues to fix in BlockHound itself.
but feel free to reopen, if it's necessary.