Support pluggable progress store about azure-event-hubs-spark

[CodingCat]

no...in direct dstream progress dir is very critical to the progress of the whole streaming pipeline...we will not open that to the user at least in the recent future

[slyons]

It's a generic ProgressTracker, in what way is having a file system necessary for the dstream approach?

[CodingCat]

because ProgressTracker is mostly about how to translate seq number to offset in the next batch which is very critical to the whole pipeline...we will not open it to the user to avoid unnecessary troubles

Similarly, Structured Streaming in Spark doesn't open offsetLog to the user and enforce it to be based on HDFS

so that's it

[slyons]

ProgressTracker seems to only manage saving the progress state back to storage and loading it back up again. It's up to the EH client to determine what the next sequence number is. The tracker is obviously important as things are blocked until the latest numbers are committed, but that doesn't mean it has to be tied to the file system.

[CodingCat]

ok, let me end the discussion,

you can implement whatever you want as the storage place for progress files. However, that does not mean we will adopt it in this project