[SUPPORT] Requesting Support for insert_overwrite in Delta Streamer about hudi

[ad1happy2go]

Thanks @soumilshah1995 for the details. can you share the full Hudi Streamer command/code which you were using?

[soumilshah1995]

its here https://github.com/soumilshah1995/DeltaHudiTransformations

[soumilshah1995]

I tried --op insert_overwrite I guess its not natively supported on docs as well it says support for insert | bulk_insert| upsert

[ad1happy2go]

@soumilshah1995 I am just thinking does insert_overwrite is even make sense for streaming workloads. Do you have any such use case?

For sources like Kafka, for sure doesn't makes sense at all.

[ad1happy2go]

The only use case I can think of is for table maintenance activity, may be run it with mode run_once.

[soumilshah1995]

Could you confirm if DeltaStreamer supports "insert_overwrite"? If not, I'm interested in understanding why. The reason for this inquiry is that in scenarios where I'm utilizing SQLSource and need to rectify an entire partition from which I'm reading, I would prefer to use "insert_overwrite" as it facilitates index lookup, akin to what "upsert" would accomplish. Ideally, having support for "insert_overwrite" in DeltaStreamer would prove immensely beneficial.

[soumilshah1995]

adding insert_overwrite can also help to build gold zone I can read data from multiple hudi tables and insert_overwrite into gold aggregated tables

[ad1happy2go]

@soumilshah1995 This makes sense. Create a JIRA also to track - https://issues.apache.org/jira/browse/HUDI-7558

[ad1happy2go]

As, Sudha suggested, can you also send a mail to dev list thread and point the conversation here. Good to hear thought on this from others.

[soumilshah1995]

Roger that

[soumilshah1995]

you want me to close this ?

[soumilshah1995]

ill send email [email protected]
ill close this thread