Comments (11)
Which revision of Sarama are you using? Specifically, does the revision you run include 396afc5 ? If it does, you may want to set producerConfig.BackPressureThresholdBytes
to match the broker's configured message.max.bytes
(which it looks like you've decreased significantly from the default - is there a reason for that?).
from sarama.
I am using latest Sarama.
message.max.bytes is set to 1000000 by default, it's not decreased.
from sarama.
Setting producerConfig.BackPressureThresholdBytes
to 1000000
(default value for message.max.bytes
) solves the problem. I can even increase this value to about 15000000
and it works (strange, needs more testing).
I wonder about two things:
- What is the difference between
producerConfig.MaxBufferedBytes
andproducerConfig.BackPressureThresholdBytes
? Especially when using asynchronous producer (QueueMessage
). - How does compression works? Does it compress single message or compress all messages in a buffer at once? It's strange that I see this problem only with Snappy or GZip enabled.
from sarama.
message.max.bytes is set to 1000000 by default, it's not decreased.
Ah, sorry, I was confusing socket.request.max.bytes
with message.max.bytes
.
MaxBufferedBytes
is how many bytes have to accumulate in order for the producer to send the current batch to the broker.BackPressureThresholdBytes
is how many bytes have to accumulate in order for the producer to stop accepting new messages entirely (until it can clear its queue). Both are only used for async (QueueMessage).- It compresses all messages at once, then has to send that as a single "message" because Kafka's protocol is kind of weird. @wvanbergen should we reduce the default back-pressure threshold to 1MB to fix this case?
from sarama.
CC also @graemej would this affect us at all?
from sarama.
@eapache thanks for clarification. If i understand correctly, MaxBufferedBytes
should be a maximum size of a batch to send and BackPressureThresholdBytes
is a buffer for all messages to send?
If yes, then I am not sure if reducing BackPressureThresholdBytes
is a good solution. I think that size of a back-pressure threshold should be much bigger than size of a single batch.
from sarama.
MaxBufferedBytes
is just a trigger saying "when I have this many bytes, send another message". BackPressureThresholdBytes
is, effectively, the maximum size of the buffer.
Perhaps adding a MaxMessageBytes
which defaults to 1MB to match kafka would be a more appropriate, though also more complex fix.
from sarama.
MaxMessageBytes
sounds good. I think it is the only way to not exceed limits defined on Kafka broker.
from sarama.
Any progress on that issue?
from sarama.
Unfortunately not, we've all been quite busy with other things and this isn't a terribly high priority for us since setting BackPressureThresholdBytes
is a functional work-around. We're happy to accept pull requests if you want to do it yourself.
from sarama.
Just merged the new producer which should fix this issue among many others.
from sarama.
Related Issues (20)
- When the Kafka broker is throttled, the producer is easily blocked HOT 5
- Support batch offset fetch requests
- What is the difference between `Consumer.Fetch.Max` and `Consumer.Fetch.Default`
- Concern Regarding the rcrowley/go-metrics Dependency HOT 2
- Module name is github.com/IBM/sarama but expected github.com/Shopify/sarama HOT 2
- Too many goroutines in gopark state HOT 1
- Error not handled HOT 2
- Error not being thrown when multiple consumer groups are created with the same group ID HOT 2
- How do I specify partitions for production messages HOT 3
- How to reset partition's offset of topic for consumer group by timestamp, use the timestamp instead of the specify offset
- Why not just set Config.Metadata.Full to false?
- NewClusterAdmin check connection
- Consumer group not reconsuming messages if message is not marked
- Consumers in a consumer group stuck for 4 days after ErrOffsetOutOfRange error HOT 4
- Sync producer erroring for repeated transactions with the same transaction ID HOT 7
- Does sarama still guarantee message ordering? HOT 1
- Sarama metrics leak in gracefull shutdown HOT 1
- [Bug]: Current offset is higher than last offset
- One bad issure HOT 2
- unaligned 64-bit atomic operation on i386 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sarama.