Add "process geyser" pattern about epocxy

[bullno1]

What's the use case for this?

[jaynel]

My intended use case is a better model for managing active Cowboy requests. Our code is currently riddled with try / catch blocks attempting to prevent a Cowboy request process from ever failing. Instead I would rather the Cowboy process monitor a worker with a timeout so that it can respond Server Busy or other Server Error on failure, and not use try / catch.

I am not a proponent of worker pools for the following reasons:

1. Failure of the worker pool causes a storm of requests to a single manager
2. Stale workers have memory issues
   • Fragmentation if the process memory isn't cleaned properly
   • The processes themselves are real memory leaks especially if they accidentally hold on to binaries
   • Can have process dictionary junk from previous executions which cause them to very occasionally fail (in ways that are hard to discover)
   • This can be caused by using 3rd party libraries which unknowingly to you use the process dictionary
3. Choosing a next worker in almost all cases ends up with a single process bottleneck
   • Even random needs to know which workers are available or risk messaging a missing process
   • Alternative worker choice algorithms can significantly impact CPU usage attempting to be independent
4. Assigning work to a limited set of processes is a premature allocation of resources
   • Standard approaches end up with large erlang mailboxes when processes are slow
   • Timeouts cause by large erlang mailboxes ripple through the pool, eventually causing full failure
   • Once allocated, the worker is expected to process, not return the work, so no reallocation occurs in most pools
5. Reporting of pending tasks is difficult because it requires all workers to anticipate query messages
   • Query messages go to the end of the mailbox, so aren't seen unless selective receive is used
6. Worker pools provide no back pressure because all task assignments are silently queued
7. Tasks can be lost when workers fail (all queued, and in transit tasks)

This proposed alternative to worker pools has the following benefits:

1. No message queues, a worker only works on a single task
   - No task loss that is unobserved
2. Workers can be more lightly managed with distributed monitors rather than a single manager process
   - No storms of restart, caller gets 'DOWN' and decides whether to request a new worker
3. Workers are never reused, so no garbage collection, fragmentation, leaks and process dictionary issues
4. If the process queue is ever empty because of resource limiting, the caller can decide what to do (back pressure)
5. Reduced garbage collection pressure, process termination (normal or otherwise) is a complete reclaim

The following issues aren't solved:

1. Reporting pending tasks, especially if the model accepts work that exceeds resources available
   - This feature may end up with an in-progress ets table indexed on worker pid owning task

This approach has the following costs:

1. Constant allocation pressure of spawning new processes, amortized by pre-spawn in tranches
   - This results in CPU pressure and memory allocation pressure
2. Pre-allocated memory for all queued processes waiting for task assignment
   - Partially offset by allowing 'hibernate' option for newly spawned processes
3. Will probably need all workers to be monitored by the Mth following process to enable resource limits
   - Complicates the coordination logic and adds a slight messaging / monitoring overhead
   - Can be avoided if resource limits are not imposed by configuration choice

I have another feature in mind (but have not created an issue yet because it isn't fully thought out) which involves "Active Tasks". It flips the worker pool concept, so that tasks are queued and processes arrive to grab them. This has the advantage of being able to report all the work pending, and potentially assigned as well, allowing concurrent reporting from central queues.

[jaynel]

Completed with v1.1.0. Didn't use queues, used a cxy_fount reservoir gen_fsm and a cxy_regulator gen_fsm, both of which are coordinated with a cxy_fount_sup supervisor.