Support HDFS sink about risingwave

[fuyufjh]

We are now working on file sink, which supports multiple kinds of storage system including AWS S3 and HDFS. cc. @wcy-fdu Can you help to link this issue to that

[wcy-fdu]

We have received your request and will support HDFS sink in the next two to three releases.

[wcy-fdu]

Hi @zhanglistar , glad to see you're interested in RisingWave. Sink to the file system is in our roadmap, could you please elaborate on your requirements for HDFS sink? For example, the sink file format/type, whether the file needs to be batched, etc.

[zhanglistar]

@wcy-fdu @fuyufjh Thanks for your reply. The background is that we want to use Risingwave to substitute Apache Flink for lower cost. There are several types of jobs running on Flink,

1. ETL, sink to Hive and data on HDFS, file format is parquet, need to be batched.
2. Java datastream API, this part is hard, And talked with @yingjunwu , no plan.
3. Flink SQL, this part is the simplest. Need to try to find how much resource can be saved from RW.
   Thanks. If you need more information, just tell me. And we can contribute the community if the thing worth doing.

[lmatz]

need to be batched.

what is the criteria for batching, by the number of rows, or by seconds?

[zhanglistar]

need to be batched.

what is the criteria for batching, by the number of rows, or by seconds?

By seconds.

[zhanglistar]

@wcy-fdu Do you plan to support Apache Hive sink in the next two to three releases?

[wcy-fdu]

We have no plans for hive sink now, but HDFS sink will be there. Contributions welcome🙌

[zhanglistar]

@wcy-fdu Looking forward to HDFS sink. Thanks a lot. We may add hive sink later.