Support unfixed backoff strategy about decaton

[ocadaruma]

Yeah, we know some users want this feature (exponential backoff) but it could be challenging because:

• In Decaton (Kafka), mixing tasks with different backoff in single partition doesn't make much sense.
  • Let's say we queue retry task with backoff-500ms to retry-topic-0
  • Next, we queue retry task with backoff-100ms to retry-topic-0
  • backoff-500ms will be consumed first. Decaton sleeps the processor-thread for 500ms till start processing
  • So, backoff-100ms will be started after processing backoff-500ms, while the backoff is 100ms.

This is inevitable because Kafka's partition behaves as queue.

Possible ideas to solve this:

• Having another internal queueing mechanism inside Decaton for different backoffs
  • However we can hold only limited tasks due to memory bound. So it doesn't scale for high-traffic or long-backoff cases
• Having different topics for backoffs
  • like retry-topic-backoff100ms, retry-topic-back-off500ms,...
  • I can find some cases on the web that achieving flexible backoff using this strategy
  • However, we need to prepare many topics

[ta7uw]

Thanks for your explanation.
It makes sense to me that supporting unfixed backoff would be challenging.