Kafka MutinyEmitter.sendAndAwait sometimes ruins blocking DB transaction. #context-propagation, smallrye-kafka, narayana about quarkus

[quarkus-bot]

/cc @FroMage (context-propagation), @Ladicek (smallrye), @alesj (kafka), @cescoffier (kafka,mutiny), @jmartisk (smallrye), @jponge (mutiny), @manovotn (context-propagation), @ozangunalp (kafka), @phillip-kruger (smallrye), @radcortez (smallrye)

[ozangunalp]

We had a similar problem with Kafka serializer touching the managed Hibernate entity during a transaction but this one isn't it. If you remove the flush and only call persist the problem goes away. That way the entity is managed by Hibernate throughout the method calla but only persisted when the transaction is committed not before.

I don't know why flush causes the context propagation to include the Kafka sender thread inside the transaction. Maybe someone with more experience with context propagation can chime in.

[mapuci]

We are dealing with legacy DB, using IDENTITY id generator strategy, which causes flush on every new insert/persist :)

[mmusgrov]

I don't know what the problem is but I can comment on the symptom: You have started a transaction in one thread which then runs some code in another thread with context propagation enabled. When you commit the transaction, presumably from the first thread, the transaction manager is telling you that the second thread is still active and is running with the transaction associated.

If you don't need the second thread to run with the transaction then use your fix by clearing the transaction context, otherwise you could wait for this second thread to return before committing the transaction.

[mmusgrov]

I notice that the transaction manager is using "onePhaseCommit" which means there is only one resource involved in the transaction which I assume is the persisting of the JPA entity so it should be fine to clear the transaction context when doing the kafka call.

[mapuci]

I notice that the transaction manager is using "onePhaseCommit" which means there is only one resource involved in the transaction which I assume is the persisting of the JPA entity so it should be fine to clear the transaction context when doing the kafka call.

I 100% agree with everything written.
I applied the ugly fix, but still, I believe that this bug should be resolved asap.
I believe that the provided scenario is quite common. Al least in imperative world; write to db, write to kafka, wait ack, commit db tx.
And having a lib, that has 1:1000 chance of wierd failure is really not great.

While having little/no experience with reactive code and context propagation... I can only guess, what is holding the mutiny-kafka-producer thread bound to thread context after acking the message.