Support router based federation about hdfs-native

[zuston]

Do you mean the RBF(Router based federation) of hdfs ?

[Kimahriman]

Yeah that's what I meant

[zuston]

Yeah that's what I meant

That's cool. This is the great project that I'm looking for, I will integrated this with apache uniffle

[Kimahriman]

Yeah that's what I meant

That's cool. This is the great project that I'm looking for, I will integrated this with apache uniffle

Thanks! Just remember it's highly experimental 😅. How would you integrate with uniffle?

[zuston]

I propose a new rust based project of riffle, which is a new rust implementation to store the shuffle data and is compatible with the uniffle original codebase.

Actually, the current riffle has supported hdfs by using the hdrs crate, which is based on the libhdfs. But this is not direct and performance is slow. So I will replace this with hdfs-native crate.

BTW, opendal is also another project that including different storage service, I think If I have time, I will push hdfs-native as alternative libhdfs choice.

[Kimahriman]

Realized router state ID tracking isn't even in a released version of Hadoop, but added support for it the best I could tell. Once it's in a released version I can try to add an integration test to verify the state ID updates