v1.9.0-rc chaos-mesh case failed about risingwave

[lmatz]

In both "create table stuck" cases, the chaos we ingested include

[ name: networkchaos-frontend-meta-longcmkf-20240511-030816, mode: one, action: partition, direction: from, duration: 600s ]

There is a network partition between the frontend node and the meta node for 10 minutes.

We expect the create table to either be successful or return an error message after 10 minutes.

[fuyufjh]

Check the await-tree dump file and didn't find anything wrong. The streaming was running normally. I guess the problem was because no one notifies the frontend create table request to stop waiting and returns an error.

There is a network partition between the frontend node and the meta node for 10 minutes.

I think the call stack will be like

frontend request (protocol layer)
  - create table handler
     - RPC call to Meta <-- waiting here

Perhaps the RPC call to Meta didn't fail as expected. Let's dive into it.

[cyliu0]

We reran those two for v1.9.0-rc-1.

One hit this again https://buildkite.com/risingwave-test/chaos-mesh/builds/829#018f7135-e456-440d-8212-0d6b161e9dee.

Another one passed https://buildkite.com/risingwave-test/chaos-mesh/builds/830.

So the reproducible rate is quite high but not 100%