Comments (5)
@13h3r The HDFS connector manages offsets itself, so I suspect you're interpreting the last part of step 2 incorrectly. You are right that offsets are committed for the consumer because the framework does this and we haven't yet exposed a way for the connector to disable it. We have a JIRA KAFKA-3462 for that and it even has a patch, but I wasn't able to review it and merge it before the 0.10.0.0 deadline.
When a task starts, it figures out what data has been delivered to HDFS (by looking at files in the directory and the WAL). Ultimately this is what tells it where to start. You can see that if it is in an initial state, it starts by trying to run a recovery process. As part of that process, it will reset the current offset to the offset where data was actually delivered.
The key to all this is that in step 1, the data is only being written to a temp file. The WAL is used to indicate the temp file should be delivered into its final location. Even if we crash between the WAL entry being written but before it is moved to its final location, the recovery process can handle completing that process.
In your scenario, the temporary data is simply discarded, but we have not actually lost any data. The next task that resumes processing that topic partition will correctly pick up where the last one left off.
from kafka-connect-hdfs.
By the way, I'd love to get a full design section into our documentation -- we'd just need to find enough time to get that written up.
from kafka-connect-hdfs.
@ewencp thanks for the detailed explanation.
But when we have no files we are unable to reset offset and messages that has been read before are lost. This can be solved by disabling offset committing to kafka, but what about providing API to commit offset from connector instead of disabling it? This maybe useful to monitor delivery lag.
from kafka-connect-hdfs.
@13h3r We have a PR in Kafka that allows disabling offset commits in Kafka apache/kafka#1139
from kafka-connect-hdfs.
Close this because of not an issue.
from kafka-connect-hdfs.
Related Issues (20)
- using wrong user/keytab while there are multiple hdfs-sink connections HOT 1
- template file isn't committed and uploaded to storage when using AvroFormat
- java.util.ConcurrentModificationException during task rebalancing HOT 1
- log4j update schedule HOT 1
- Hive table does not match column names present in the parquet data
- Exception when reading Decimal types written by connector
- Hive Merge Feature
- Incremental Co-operative Rebalancing Support for HDFS Connector
- Error after install and unistall connect-transforms
- Adding Hive partition threw unexpected error
- HDFS2 connect compatibility with HDFS3 server
- CVE-2021-34538 HIGH vulnerability HOT 2
- Task is being killed and will not recover until manually restarted
- Allow to limit retry write errors by timeout
- Kafka Issue while running on docker and adding new connector HOT 1
- can't build because repo conjars is down
- multiple keytab kerberos issue HOT 1
- OzoneFileSystem
- Non-resolvable parent POM io.confluent:common:[7.7.0, 7.7.1) for io.confluent:kafka-connect-storage-common-parent:11.2.9
- [2024-05-30 10:25:31,403] ERROR [hdfs3_sink-test_v4|task-0] WorkerSinkTask{id=hdfs3_sink-test_v4-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:237) java.lang.NullPointerException: Cannot invoke "io.confluent.connect.hdfs3.DataWriter.open(java.util.Collection)" because "this.hdfsWriter" is null
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-hdfs.