Timely worker synchronization for new dataflow creation about materialize

[frankmcsherry]

Btw, it is probably worth spec'ing out which parts of Materialize exist in which offerings. My sense is:

1. There is a very reasonable, non-durable, non-autoscaling, non-clustermanaged thing that each user could try out on their laptop / in a browser / etc., which is a simple binary download without GBs of Apache cruft, and which gets them up and running in minutes.
2. There is also a very reasonable durable, autoscaling, cluster managed thing that wraps around this that is worth paying for, and that serious people would agree that they need, even though the above did not crash during their demo runs.

That's one hypothetical partitioning of value, but with something like that in mind we can more clearly determine which features need to go where. At the moment interactive is the closest thing to (1.) above, but it isn't obvious (to me) that we want to add all of the features in to it, both because they may make the initial experience more awkward, and because it may be "useful" to hold them back.

[rjnn]

Currently, @benesch is working on an intermediate API, 'metastore'. The intention is for metastore to be backed by zookeeper for (2). But it's equally possible to have a shim 'zookeeper' that's a linked-list that satisfies (1) with no additional Apache cruft.

The reason why zookeeper is such an attractive option is that if the primary streaming data ingest layer is Kafka, then Kafka users typically have Zookeeper running anyhow.

[benesch]

Closing this out. I'm pretty happy with the end result, which uses ZooKeeper to store the metadata, but then the Chosen Node (worker node 0) pushes them through a timely sequencer for a consistent ordering.

[benesch]

For posterity: getting ZooKeeper to expose a consistent stream of events is literally impossible. It's not built for that. I think it might be possible with etcd, but even if it is, it's definitely awkward. If we want that, we'd need to bundle a Raft implementation, and that seems like serious overkill. The solution of using ZooKeeper for persistence and timely's sequencer for ordering actually seems like the best option.