Comments (15)
Should be fixed by #17
from jruby-kafka.
Maybe I'm misunderstanding, but that call is only made if reset_beginning is true. The maybe delete offset is only successful if no other consumers within that group are processing data. In normal situations you would leave reset_beginning to false and the consumer will resume at its group's last offset or the smallest offset available if the topic has since purged that offset in cases such as the data got too old.
from jruby-kafka.
Sorry I was confusing my libraries. You shouldn't be including the config option reset_beginning for a consumer group unless you want it to reset. If you don't include the option your consumer group should resume where it was last committed to zookeeper. My explination about the offset selection otherwise is still valid.
from jruby-kafka.
This is what Kafka's docs say about the setting:
What to do when there is no initial offset in ZooKeeper or if an offset is out of range:
- smallest : automatically reset the offset to the smallest offset
- largest : automatically reset the offset to the largest offset
- anything else: throw exception to the consumer
from jruby-kafka.
I guess it isn't clear how ZkUtils.maybeDeletePath
works. From the code at https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/utils/ZkUtils.scala#L449 it seems like it would succeed whenever it was called if the path existed. It seems like jruby-kafka would call ZkUtils.maybeDeletePath whenever a group is run with @auto_offset_reset == 'smallest'
, so wouldn't the offsets be deleted if I set auto_offset_reset to 'smallest' and run? If not, what causes maybeDeletePath to fail so that the offsets in ZK are retained so the consumer can start where it left off?
from jruby-kafka.
This appears to be similar to an issue with the Spark Kafka consumer: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-and-the-spark-shell-td3347.html#a3387
Here is the commit that addresses the issue in Spark's consumer that appears to relate to this issue:
apache/spark@c8850a3
Again, auto_offset_reset="smallest" should only cause the consumer to start at "the topic beginning" if there is no offset stored in ZK or the stored offset is out of range. If a consumer goes offline, and comes back online with auto_offset_resets="smallest" it should normally start at its last offset, not have the offset deleted.
I want to reference the next-to-last bullet point in the "Known issues in Spark Streaming" section of this:
http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/
from jruby-kafka.
Can you write a test to repeat the issue as I'm having a hard time
replicating it.
On Nov 21, 2014 3:27 PM, "Sean McKibben" [email protected] wrote:
This appears to be similar to an issue with the Spark Kafka consumer:
http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-and-the-spark-shell-td3347.html#a3387Here is the commit that addresses the issue in Spark's consumer that
appears to relate to this issue:
apache/spark@c8850a3
apache/spark@c8850a3Again, auto_offset_reset="smallest" should only cause the consumer to
start at "the topic beginning" if there is no offset stored in ZK or the
stored offset is out of range. If a consumer goes offline, and comes back
online with auto_offset_resets="smallest" it should normally start at its
last offset, not have the offset deleted.I want to reference the next-to-last bullet point in the "Known issues in
Spark Streaming" section of this:http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/
—
Reply to this email directly or view it on GitHub
#16 (comment)
.
from jruby-kafka.
I will try to come up with some way to reproduce, but it is one of those concurrency issues that is difficult to get to happen reliably. It only seems to occur in rebalancing or other cases when multiple threads/processes are part of the same consumer group. But when the group's ZK record does get deleted, it can be pretty disastrous, as accidentally starting from the very beginning of a big kafka topic during a rebalance that could have itself been triggered by a worker OOMing can cause a cascading failure of another worker on a topic, causing more rebalancing and more OOMing and more ZK metadata deletion.
I think this issue explains things most clearly:
https://issues.apache.org/jira/browse/SPARK-2492
Especially the differences between 0.7 and 0.8 with regard to autooffset.reset
vs auto.offset.reset
vs the console --from-beginning
option.
Seems like a best practice to try to stick to just wrapping the official High-Level Consumer rather than introducing any additional ZK operations in the ruby code.
from jruby-kafka.
Okay I've finally seen the light here. I'll accept the PR but am going to change it a bit to match how Kafka implemented fixing https://issues.apache.org/jira/browse/KAFKA-1431 apache/kafka@b866c55
Will try to do this soon.
from jruby-kafka.
Thanks for raising this @graphex it should be fixed now.
from jruby-kafka.
Glad to hear, will be putting it to the test soon! Thank you.
from jruby-kafka.
Thanks Joe for the fix. I am facing the same issue. When is the next release planned with the latest fix?
from jruby-kafka.
I pushed out 1.1.0.beta just now.
from jruby-kafka.
Thanks a lot Joe. Is that release from the latest commit in master? https://github.com/joekiller/jruby-kafka/commits/master
from jruby-kafka.
I pushed out 1.1.0 as non-beta this afternoon. https://github.com/joekiller/jruby-kafka/releases/tag/v1.1.0 and https://rubygems.org/gems/jruby-kafka/versions/1.1.0-java
from jruby-kafka.
Related Issues (20)
- Consumer parallelism? HOT 7
- Where can I find group.rb? HOT 1
- Latest gem release (for jruby) doesn't have new, simpler consumer. HOT 2
- Create CI build
- Are there any plans to upgrade to the latest Kafka API? I would like mutual TLS support HOT 6
- jruby build dependencies
- Implement KafkaConsumer
- Add CI for v1_x HOT 1
- add LZ4 codec for producer
- 3.0 - Cannot get past iterator.hasNext HOT 9
- Namespace conflicts HOT 3
- Use Kafka Clients Jar
- java.lang.NoClassDefFoundError: Could not initialize class org.apache.kafka.clients.producer.ProducerConfig HOT 6
- Port CI release to v1_x branch. HOT 1
- log4j warnings HOT 2
- Consumer stops working HOT 6
- ArgumentError: Parameter :metadata_broker_list is required.
- avro support
- Use kafka-client 2.0.0 jars HOT 2
- jruby-kafka_jars.rb missing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jruby-kafka.