[ddc] flaky behavior in for-loop eval test about sdk

[sigmundch]

After testing a couple ideas, I stumbled upon the cause of this issue. (I added a print to validate that our breakpoint was not improperly added to the increment statement. In doing so, the failure logs revealed what follows...)

In a chain of changes we sped up expression evaluation tests by combining all tests into a single input program and reusing the same page across all test cases. This made a big reduction in timeouts and made tests complete a lot faster (sometimes 5x or more). This came at a cost of loosing some isolation.

Turns out, we have a race when we advance from one test case to the next. We:

1. (test 1) set a breakpoint
2. (test 1) start main
3. (test 1) run all checks at a breakpoint
4. (test 1) remove the breakpoint
5. (test 1) resume execution (running the rest of the test and the rest of main)
6. (test 1) move on to the next test set up, that is:
7. (test 2) set a breakpoint
8. etc

Step 6 & 7 are done by our test driver in separate process from the page. So they may complete before step 5 finishes running.

This is what trips the loop-variable test, which was the second test case using a breakpoint inside a loop.

When the previous test finished and resumed execution, the loop continued running and the breakpoint of the second loop test ends up being hit by the from from the first test. This led to broken assumptions about the state of the loop in the second test case.

A few options to bring some isolation between tests are:

• a. Don't reuse any breakpoints across tests.
• b. Ensure execution is done before step 6
• c. Don't reuse reentrant breakpoints.

(b) is requires some infra in the test driver to wait for a signal that main finished running. This may mean monitoring that some state is updated on the page and ping the page periodically for updates until the state changes. Sounds doable but may have a performance cost that we'd like to avoid.

(a) brings isolation because the location where one test stopped is less likely to affect how another test behaves. That siad, we have many test cases that do reuse the same breakpoint for simplicity and turns out that most of them are not reentrant: that makes them in a way isolated by the fact that once resumed the breakpoint cannot be hit again.

This leads me to believe that (c) may be a simpler approach. It behaves like (a), but only requires work on tests containing loops (currently there is only 1 case of a loop breakpoint reused across 2 tests).

[sigmundch]

Hopefully fixed by 4e0ece2