Comments (13)
I'm going to be out for the next few days. @rdhabalia can you take a look at this?
from pulsar.
Also, redeliverUnacknowledgedMessages(List<MessageIdImpl> messageIds)
doesn't seem to be removing redelivered messages from the queue at any time, so the queue will always be stuck with redelivered messages.
I can't think of any simple way of solving this right now, maybe we can keep track of messages in the queue on a separate set.
from pulsar.
redeliverUnacknowledgedMessages : it cleans up all the messages present into incomingMessages
and sends separate flow-message
command which increments availablePermits
at broker for a given consumer. So, consumer will not be stuck at any time.
However, right redeliverUnacknowledgedMessages doesn't clean up incomingMessages
queue and asks broker for redelivery. So, as soon as client consumes N/2 messages from the incomingMessages
queue it triggers flow-message
command to broker and broker will first release those redelivered messages as part of received permits. In this case: again consumer will not be stuck.
Also, redeliverUnacknowledgedMessages(List messageIds) doesn't seem to be removing redelivered messages from the queue at any time
Yes, we have to make similar change of redeliverUnacknowledgedMessages to clean up from incomingMessages
queue and sends permits to broker by triggering flow-message
command.
from pulsar.
I will make changes and add test-cases for this scenario.
from pulsar.
@rdhabalia I just created a PR for this, please give it a look, perhaps you can think of a better way of approaching this.
from pulsar.
Ok. thanks, I will look into it.
from pulsar.
@merlimat @rdhabalia any news on this? We really need to solve this issue
from pulsar.
@sschepens just got back in office. reviewing the PRs
from pulsar.
Just to clarify, this issue was introduced when tracking timeout on messages not pulled from the queue.
Perhaps we can take a different approach to this, in the end what we want to prevent is a clientes to hold messages when it's not processing.
We could empty the queue and not as for messages until it resumes acknowledging messages or something of the sort.
from pulsar.
OK, it looks like it would be easier to just empty the queue and have the broker resend the all the messages that were on it. Anyway, they're not going to be "duplicates" since the application would not have seen them already.
from pulsar.
Yup, but we maybe should make the consumer not receive messages until it resumes acks. Just to prevent constant redelivery.
from pulsar.
I agree on that, the library should be playing with the permits to do that, but I'd keep it separate from this PR, because:
- It might be tricky to make sure we don't forget any corner case and leave the delivery stopped
- It should not be super-awful anyway.. you get 1000 messages resent every n-minutes
from pulsar.
Fixed in #101
from pulsar.
Related Issues (20)
- Deleting the Partitioned topic still available in the topic list
- Suddenly the built in mongo connector stopped the source
- Duplicate message created using partitioned topic with flink
- [Bug] [cli] Pulsar-client cli doesn't support timeout values given in apache pulsar reference website
- [Bug] [cli] Pulsar Tokens Create is mishandling time units (specifically, treating seconds as milliseconds) HOT 1
- Excessive memory allocation in OTel broker metrics HOT 3
- [Bug] pulsar-admin 2.x links don't properly link to 3.2 HOT 6
- [Bug] Offload to S3 triggered manually returns success, while not uploading HOT 2
- [Bug] Broker became irresponsive due to deadlock during race-condition in metadatastore callback
- [Bug] Broker is failing to load stats-internal with broken schema ledger
- [Doc] Incorrect description of UniformLoadShedder in pulsar site. HOT 1
- [Doc] PIP-356: Improve "Support Geo-Replication starts at earliest position" doc
- [Bug][client] Consumer implementation might change message processing order when ack timeout is set
- High GC pause cases high publish latency HOT 3
- Flaky-test: ExtensibleLoadManagerImplTest.testGetMetrics (fails consistently)
- [Bug] bin/pulsar-zookeeper-ruok.sh fails with apachepulsar/pulsar:3.3.0 image
- [Bug] maven build fails with Java 22 HOT 1
- [Bug] nslookup in apachepulsar/pulsar:3.3.0 isn't compatible with kubernetes search domains
- Pulsar Standalone: --wipe-data does not work with RocksDB backend in 3.2.3
- [Doc] Search doesn't work on pulsar website HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulsar.