Modify Offsets committed to next expected offset instead of current consumed offset about faust

[StephenSorriaux]

This explains why some tools always reports a lag of 1 for every faust consumer group. Thanks for this issue.

[alexpirine]

That's so weird. I find it hard to believe faust developers just didn't notice this convention.

Would be very interested to know the reason why offsets are committed this way: I believe there must be some reason.

[vineetgoel]

We did notice this. I believe when we started working on this, we wanted to be more consistent with how most message queues manage offsets. Later on when we doubled down on our use of Kafka, we realized this was a bit of a mistake and have a it in our backlog to make this more consistent with Kafka.

[enavarro222]

Is it any new on that point ?
This is very annoying if one want to use burrow with faust consumers !

Indeed burrow raise an alarm if a consumer lag stay greater than 0 for a given time window.

[sivasai-quartic]

Hi, I'm also facing the same issue. Like others said, it's already processed the last record, and it's not updated the offset. but its not processed the last record at all. I have a total of 8 records that are available in Kafka, but only 7 records are being processed.

here is the sample code:

@app.agent(input_topic, concurrency=1)
async def my_task(tasks):
    async for my_task in tasks.take(record_per_partition, within=poll_interval):
        assert len(my_task) > 0
        asyncio.gather(*(process_payload(json.loads(args.decode('utf-8'))) for args in my_task))

felt like, its a high priority.