Channel stop receiving about jox

[adamw]

Thanks for the update :) I did run an overnight test as well, but didn't find anything.

Maybe you are experiencing thread pinning, as part of your blocking call? This can be traced by adding -Djdk.tracePinnedThreads=full (see e.g. https://todd.ginsberg.com/post/java/virtual-thread-pinning/)

[adamw]

Hm that's very suspicious. No, there's nothing time-based in the code. I know it's hard to reproduce, but the Channel.toString dumps a lot of internal representation information, which might help to debug such issues.

Is there some specific pattern of production/consumption that you are using, that might help to reproduce the problem?

[svenkubiak]

Let me try the Channel.toString. I have also added a check for isClosedForReceive() - was only checking isClosedForSend so far.

Using jox more or less out of the examples with a little wrapper around it - see:
https://github.com/svenkubiak/mangooio/blob/feature/8.5.0/mangooio-core/src/main/java/io/mangoo/async/EventBusHandler.java

[adamw]

Indeed this seems like rather ordinary usage

In addition to the testsuite (which includes benchmarks sending many messages), I'm also running a simple two-thread send-receive loop locally, so far 47298000000 messages have successfully been sent & received ;)

If you'd have some more information, that would be great.

[svenkubiak]

I am quite sure that this is caused by the virtual thread that my receive() call is wrapped in. Looks like this is not intended for a blocking call. I have made some adjustments in my code and going to run further tests. But I am absolutely convince that this it not related to jox at all.

[svenkubiak]

While I was quite certain that switching from a virtual thread to a platform thread would fix my issue, unfortunately it didn't. Stopped receiving after around 12 hours as before. While I was further investigating this I noticed that in the doReceive() method there is a check regarding the segment id

if (segment.getId() != id)

If this would match equal, then doReceive would end up in an infinite loop not receiving. What would be the situations when this could happen?

As this is not mentioned in the readme, your recommendation for have a constant received would be to put the receive() in an infinite loop, right? So something like this:

Thread.ofVirtual().start(() -> {
    try {
        while(true) {
                var payload = channel.receive();
                //handle payload
        }
    } catch (Exception e) { //NOSONAR
        ...
    }
});

I have adjusted my implementation again and doing additional tests now.

[adamw]

If you are referring to this line, on the next iteration it will re-read the receiveSegment value, and it should already be move forward. This might only happen if there are concurrent receives happening - which I don't think is the case here (when another recevie() moves forward the segment). The segment is moved forward in findAndMoveForward.

Yes, the loop looks right. Maybe try catching & logging a Throwable, so that e.g. OutOfMemoryError is also discovered? Or better, installing an uncaught exception handler.

Did you try capturing the stack trace?

[svenkubiak]

If you are referring to this line, on the next iteration it will re-read the receiveSegment value, and it should already be move forward. This might only happen if there are concurrent receives happening - which I don't think is the case here (when another recevie() moves forward the segment). The segment is moved forward in findAndMoveForward.

Yes, the loop looks right. Maybe try catching & logging a Throwable, so that e.g. OutOfMemoryError is also discovered? Or better, installing an uncaught exception handler.

Did you try capturing the stack trace?

Yes, I did. And after a lot of digging, it turned out to be an OutOfMemoryError Exception in an InfluxDB connection which was part of the what the receive() was handling. That was one of the stranges behaviours I have seen. The whole application kept behaving normal in some parts.

Thanks for your support!

[adamw]

Ah, good to hear it's resolved :)