Comments (5)
@allxone The connector can't actually support this directly itself. We could effectively have it exit by triggering an error that causes the framework to shut it down. However, it would be marked as failed and the connector itself would continue to exist. To do this correctly would also require framework level support for connectors that have a definitive ending point. But this is also antithetical to Connect's design, so I'm not sure it's something we'd want to support. Is there a reason you wouldn't just use the connector as is and set the file rotation interval to match your desired frequency?
from kafka-connect-hdfs.
@ewencp thanks for the clarification.
I understand my request could result in an anti-pattern against streaming ingestion standards and this is confirmed also by the fact that I wasn't able to find any tool with the characteristics I'm looking for.
My requirement is reducing the number of moving parts out of the Oozie workflow control: if the HDFS connector is down the workflow would simply miss data without recognizing the issue.
Being able to specify at the connector startup the ending offset after which it would exit gracefully would be perfect for my use case. With that option I could preliminary query Kafka for the latest topic offset and start the connector knowing it will always exit as soon as it finish consuming the latest messages.
BR
Stefano
from kafka-connect-hdfs.
@allxone Yeah, usually where we've seen this request is someone trying to map an existing batch workflow to Connect.
We don't expose the right metrics for this currently (at least not via Connect, you could get them from the internal producer and consumer we use), but the other way to accomplish this would be to start the connector and then just wait for it to hit a steady state of processing 0 msg/s. I realize that might seem like a bit of jumping through hoops compared to just having the connector stop, but it would work for any connector and doesn't require any support from connector or framework (beyond stuff that we want to support anyway).
from kafka-connect-hdfs.
Hi, could you give a little more insight, or some direction to some documentation, on how I might do this:
"...but the other way to accomplish this would be to start the connector and then just wait for it to hit a steady state of processing 0 msg/s."
from kafka-connect-hdfs.
closing as this has been provided a workaround
from kafka-connect-hdfs.
Related Issues (20)
- Issue in a Kerberized environment a day after renew ticket HOT 2
- Explain limitation listed in the documentation HOT 3
- using wrong user/keytab while there are multiple hdfs-sink connections HOT 1
- template file isn't committed and uploaded to storage when using AvroFormat
- java.util.ConcurrentModificationException during task rebalancing HOT 1
- log4j update schedule HOT 1
- Hive table does not match column names present in the parquet data
- Exception when reading Decimal types written by connector
- Hive Merge Feature
- Incremental Co-operative Rebalancing Support for HDFS Connector
- Error after install and unistall connect-transforms
- Adding Hive partition threw unexpected error
- HDFS2 connect compatibility with HDFS3 server
- CVE-2021-34538 HIGH vulnerability HOT 2
- Task is being killed and will not recover until manually restarted
- Allow to limit retry write errors by timeout
- Kafka Issue while running on docker and adding new connector HOT 1
- can't build because repo conjars is down
- multiple keytab kerberos issue HOT 1
- OzoneFileSystem
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-hdfs.