Proposal: streaming, incremental computing about bigslice

[cosnicolaou]

I think I can envision how this would work in practice, namely, I can tradeoff the efficiency of processing larger batches to gain more streaming/pipelining, which I think is totally appropriate. How would operations like cogroup work though? Can they still be incremental, I would imagine not?

[mariusae]

I think I can envision how this would work in practice, namely, I can tradeoff the efficiency of processing larger batches to gain more streaming/pipelining, which I think is totally appropriate. How would operations like cogroup work though? Can they still be incremental, I would imagine not?

They can be: cogroup would checkpoint its state as, e.g., a mapio sstable; it can then emit rows for any newly modified keys.

There is also the opportunity, later, to introduce a notion of a "diff+patch" approach, where operators emit fine-grained changes that can be reconstructed downstream. But this requires a fairly different data model, though it could nevertheless be built on top of this proposal.