Expected Behavior on Startup? about broadway

[josevalim]

latest will get the latest entry before the consumer group offset is committed. If you want to consume what was there before the consumer group started, you want earliest. :)

[yordis]

Hey @josevalim watching the following video https://youtu.be/ovdSOIXSyzI from Confluent peeps at 6:48; he mentioned the following:

6:47: If the consumer group instance is restarted,
6:48: the first thing it can do is to make a request
6:52: to the coordinator to retrieve its last committed offset.
6:55: And once it has the offset,
6:56: it can resume the consumption
6:59: from that particular offset.
7:00: If this consumer instance is started for the very first time
7:03: and there's no saved position for this consumer group,
7:07: then you can either start consuming
7:10: from the very beginning of this topic partition
7:13: or from the very latest offset.

Mainly the following (I am highlighting a few keywords with asterisks)

7:04 If this consumer instance is started for the **very first time**
7:03 and there's **no saved position for this consumer group**,
7:07 then you can either start consuming
7:10 from the very beginning of this topic partition
7:13 or from the very latest offset.

Reading your comment you mentioned the following:

If you want to consume what was there before the consumer group started

Did you explicitly mean to say started but exclude restarted?

I am trying to understand the expected behavior here because my understanding is that based on what the video is saying after a Consumer Group offset is committed, the value of offset_reset_policy doesn't matter anymore because an offset was committed. It would matter only the very first time the consumer group starts.

Right?

[yordis]

Kind of related to dashbitco/broadway_kafka#27, it said that it was fixed.

We are observing the following.

1. We started the consumer with :latest and processed and commit the offset
2. We changed to be :earliest
3. We re-consumed the messages

I would expect the offset_reset_policy to be ignored once some consumer group offset is committed.

[josevalim]

I would expect the offset_reset_policy to be ignored once some consumer group offset is committed.

Me too.