Comments (5)
This explains why some tools always reports a lag of 1 for every faust consumer group. Thanks for this issue.
from faust.
That's so weird. I find it hard to believe faust developers just didn't notice this convention.
Would be very interested to know the reason why offsets are committed this way: I believe there must be some reason.
from faust.
We did notice this. I believe when we started working on this, we wanted to be more consistent with how most message queues manage offsets. Later on when we doubled down on our use of Kafka, we realized this was a bit of a mistake and have a it in our backlog to make this more consistent with Kafka.
from faust.
Is it any new on that point ?
This is very annoying if one want to use burrow with faust consumers !
Indeed burrow raise an alarm if a consumer lag stay greater than 0 for a given time window.
from faust.
Hi, I'm also facing the same issue. Like others said, it's already processed the last record, and it's not updated the offset. but its not processed the last record at all. I have a total of 8 records that are available in Kafka, but only 7 records are being processed.
here is the sample code:
@app.agent(input_topic, concurrency=1)
async def my_task(tasks):
async for my_task in tasks.take(record_per_partition, within=poll_interval):
assert len(my_task) > 0
asyncio.gather(*(process_payload(json.loads(args.decode('utf-8'))) for args in my_task))
felt like, its a high priority.
from faust.
Related Issues (20)
- GlobalTable not working when using multiple instances
- [Question] Get consumer offset/lag of each partition from the producer side
- Missing `f` prefix on f-strings
- Make Faust compatible with Python 3.9+ for production environment (rocksdb: faust-streaming[rocksdb] ) on MacOS Monterey 12.3.1 HOT 1
- Update dependency to `click>= 8.0` HOT 3
- Is Faust died? HOT 7
- Robin
- Support for python3.10 HOT 1
- A Faust Alternative - Fluvii
- Dagger - A Faust Based Workflow Engine
- faust overrides global logging settings -- unable to use structlog HOT 2
- Couldn't set Retry Topic mechanism In Faust Streaming HOT 1
- kafka-ConnectionError in test HOT 1
- Consumer crashes sporadically and stops consuming - Crashed reason=ConsumerStoppedError()
- asyncio.exceptions.CancelledError and consumer stops HOT 1
- How can i join two topics in stream?
- ANN: Please migrate away from pytest-openfiles
- Faust Error: Address In Used
- Does Faust support the "Timestamp Extractor" feature? HOT 1
- What could be causing inconsistent message processing in a Kafka-based system used to synchronize data between MSSQL and MySQL databases, despite successful message delivery to the Kafka topic
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faust.