Misbehaving driver can cause Fluid to hang on container open about fluidframework

[scarlettjlee]

FYI, @rajatch-ff

[kashms]

@zagriswo we'll improve the checks here, work backlogged.

[jatgarg]

There is no way for the container to progress until it fills the gap with the ops because the state would not be consistent then. We cannot skip ops and just proceed.

[zagriswo]

Agreed on not "just proceed" but there should be some sort of additional check that results in load erroring out. If the driver returned success but no messages, should that be an immediate error? Should there be a limit on how many attempts are made for the same gap?

[jatgarg]

Agreed on not "just proceed" but there should be some sort of additional check that results in load erroring out. If the driver returned success but no messages, should that be an immediate error? Should there be a limit on how many attempts are made for the same gap?

Question: In your original description, why did the service kept returning 0 messages and made the container to stuck? Did it ever proceed, or the service lost the messages somehow?

[zagriswo]

@jatgarg it was a bug uncovered by fuzzing. Basically, a hole was made in the op stream, so our driver returned 0 messages in perpetuity because those messages just didn't exist anymore.

[jatgarg]

In ODSP driver, we already handle this issue where if we don't make progress in fetching ops using delta storage service, then we give up after 30 secs and container closes.
We use this public utility api for that: requestOps()

FluidFramework/packages/loader/driver-utils/src/parallelRequests.ts

Line 545 in e9ba4de

export function requestOps(

You can see the usage of it in ODSP driver here:

FluidFramework/packages/drivers/odsp-driver/src/odspDeltaStorageService.ts

Line 225 in e9ba4de

const stream = requestOps(

Let me know if you have more questions. You should be able to use it with your driver easily.

In future, we will think if we want to move this thing higer up the stack and in loader/deltastream layer.