Proposed RFC Feature: Automatically Discover Intermittently Failing Tests about sig-testing

[jckand]

1. It would be great if we had some way to notify the actual test owners with this data, rather than only the person submitting an AR change so it's more actionable data.
2. Timeout may need to be adjusted if a large amount of Python tests fail, due to Editor/AP spin-up time. May just require experimentation.

[robhaco]

On the technical side, it will likely be a requirement to create some utilities related to managing/monitoring the pytest cache due to what appears to be common issues with the cache needing to be invalidated/reset to get the expected results from the --last-failed argument.

[AMZN-Dk]

Per the SIG testing meeting on 1-28-2022 this RFC has been approved to more forward pending the implementation of a test failure resolution policy per #26

[jckand]

On the technical side, it will likely be a requirement to create some utilities related to managing/monitoring the pytest cache due to what appears to be common issues with the cache needing to be invalidated/reset to get the expected results from the --last-failed argument.

Good callout. I believe each new feature "suite" that runs has its own pytest cache, so this may need to cache and check all of them.

[Kadino]

Needs to be informed by the policy generated by #26

[evanchia-ly-sdets]

Duplicate of #43