Comments (6)
I used fieldPartioner to handle it. But i think your idea is better!
from kafka-connect-hdfs.
Agree that timeBased should have the option to use event time , for example event that send from mobile SDK at 8:58 might be pushed to kafka at 9:01 , using the ingestion time is not accurate
I also added pull request in order to support several fieldsPartitioner as in my use case I need both event time and event type in order to partition the events in parquet
from kafka-connect-hdfs.
@xianzhen FieldPartitioner
only works for exact values, which means you can't use a timestamp field directly, you need it to be simplified to, e.g., minute, hour, or day, depending on the granularity you want.
@KerKerO I think this would definitely be valuable. In particular, Kafka added timestamps natively in its format (but after this connector was originally created) so we should at least support those. Additionally, since Connect supports timestamp logical types, we should definitely have support for those fields. I think this should amount to a small additional config to specify the field and minor modifications to TimeBasedPartitioner
.
from kafka-connect-hdfs.
Agree that FieldPartitioner cant use timestamp , in my use case each event that i am sending to kafka contains also the following fields : day like 2016-08-23 , time like 10:00
this fields are calculated from the event timestamp (I am adding day/time fields to the event that i am sending to kafka)
from kafka-connect-hdfs.
A little head-up, it seems that kafka-connect-hdfs has been change to use kafka-connect-storage-common which does have a very nice way to provide a timestamp field ( it can be seen here in kafka-connect-s3 which also use kafka-connec-storage-common. We now have to wait for this change come into release :) From what i see in the branch, 3.3.1-alpha2 still does not have mentioned change applied :(
from kafka-connect-hdfs.
Indeed, timestamp-based partitioners where timestamp is stored in the Kafka record or a record field have become available under kafka-connect-storage-common
.
Two recent PRs have added support of such partitioners in the HDFS connector (#245 and #246).
Closing this issue, and we may reopen this one or a new one if there's an issue with these partitioners. Anyone interested in field-based timestamp partitioning is highly encouraged to try what's on master right now! Thanks everyone for highlighting this feature!
from kafka-connect-hdfs.
Related Issues (20)
- using wrong user/keytab while there are multiple hdfs-sink connections HOT 1
- template file isn't committed and uploaded to storage when using AvroFormat
- java.util.ConcurrentModificationException during task rebalancing HOT 1
- log4j update schedule HOT 1
- Hive table does not match column names present in the parquet data
- Exception when reading Decimal types written by connector
- Hive Merge Feature
- Incremental Co-operative Rebalancing Support for HDFS Connector
- Error after install and unistall connect-transforms
- Adding Hive partition threw unexpected error
- HDFS2 connect compatibility with HDFS3 server
- CVE-2021-34538 HIGH vulnerability HOT 2
- Task is being killed and will not recover until manually restarted
- Allow to limit retry write errors by timeout
- Kafka Issue while running on docker and adding new connector HOT 1
- can't build because repo conjars is down
- multiple keytab kerberos issue HOT 1
- OzoneFileSystem
- Non-resolvable parent POM io.confluent:common:[7.7.0, 7.7.1) for io.confluent:kafka-connect-storage-common-parent:11.2.9
- [2024-05-30 10:25:31,403] ERROR [hdfs3_sink-test_v4|task-0] WorkerSinkTask{id=hdfs3_sink-test_v4-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:237) java.lang.NullPointerException: Cannot invoke "io.confluent.connect.hdfs3.DataWriter.open(java.util.Collection)" because "this.hdfsWriter" is null
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-hdfs.