[BUG][Search Backpressure] High Heap Usage Cancellation Due to High Node-Level CPU Utilization about opensearch

[kaushalmahi12]

The backpressure works as follows and here the heap_domination threshold is mere 0.05 percent of total jvm memroy available for the process. The same flowchart is applicable for both SearchTasks and SearchShardTasks

There are basically three trackers which can potentially cancel a task

• Heap Based
• CPU Based
• Time Based
  The basic flaw in this is even when node is not under duress because of Heap, the tracker will still kick in and cancel the task
  

[jainankitk]

Assigning to @kaushalmahi12, due to his prior context with query sandboxing and search backpressure

[sohami]

@kaushalmahi12 Agreed and that is what the issue is trying to explain. I think we should check the duress condition for each tracker as well. For example: If under heap duress, then only evaluate the task for heap based cancellation.

[kaushalmahi12]

Thats right @sohami
The weird thing about this is that, it takes the tasks for cancellation even when total jvm allocations by co-ordinator/shard level tasks are 0.05%. ref.
Time Based cancellation is still not justified for cases where the cluster has very light search traffic and user is fine with higher latencies for those queries (Given only CPU is high and AC is already there to safeguard against new incoming requests).

I think we should increase this threshold for search workload JVM(or remove it altogether) and separate out the corresponding trackers.

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@ticheng-aws Thanks for creating this issue, looking forward to seeing this resolved.