Use Kafka log compaction in update queue to intelligently drop old models/updates about oryx

[srowen]

Hm, on reading into this more, I'm not sure it's applicable, unfortunately. As https://issues.apache.org/jira/browse/KAFKA-631 says, it is for data with primary keys of some kind, and of course for data that is updated.

Here, there is no primary key for any of the model or update data, and it's only the MODEL key that is conceptually updated. A unique ID can be assigned to updates, but this will mean the compact process spends most of its time compacting records that can't be compacted. Without unique IDs, almost all updates would be thrown away, which can't be done.

If it were easy to only compact MODEL, this would work. Or, to somehow always compact up to the most recent MODEL key.

For now the idea is to set topic retention to something not too long compared to the generation interval. For example maybe at most 3 days if building a model once a day.