v1.7.0-rc/nightly-20240201 source throughput down to 0 with non-shared PG CDC sources about risingwave

[cyliu0]

Reran the queries except q3 with v1.7.0-rc-1 passed https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/199

[StrikeW]

Update: revert #14899 also reproduce the problem, investigating other PRs in the list.

[StrikeW]

I found that the stream query in the passed job generated much less data compared with failed jobs.

And there is join amplification in the failed jobs:

I suspect the workload has changed and I rerun the pipeline with nightly-20240131 again, then it also experience barrier piled up as those failed jobs.

So I think the pipeline failure is not caused by the code change. cc @lmatz if you have other information.

[lmatz]

thanks for the findings, let us check if there is any changes on the pipeline side

[cyliu0]

Recently, I've added the ch-benchmark q3 back to the pipeline. The q3 has been removed since #12777

So I think it's still a problem?

[cyliu0]

Hit again for v1.7.0-rc-1 with following queries

CH_BENCHMARK_QUERY="q1,q2,q4,q5,q6,q9,q10,q11,q12,q13,q14,q15,q16,q17,q18,q19,q20,q22"

https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/201
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&from=1707121771852&to=1707137671126&var-datasource=ebec273b-0774-4ccd-90a9-c2a22144d623&var-namespace=ch-benchmark-pg-cdc-20240205-085511&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All

[cyliu0]

Hit again with nightly-20240207

https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/205#018d8597-b39f-48ed-bbea-9f1b27b911c8

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=ebec273b-0774-4ccd-90a9-c2a22144d623&var-namespace=ch-benchmark-pg-cdc-daily-20240207&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All&from=1707342785684&to=1707348581400

[lmatz]

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=ebec273b-0774-4ccd-90a9-c2a22144d623&var-namespace=ch-benchmark-pg-cdc-daily-20240207&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All&from=1707342785684&to=1707348581400

Why the compaction duration is so high?

[StrikeW]

Hit again with nightly-20240207

https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/205#018d8597-b39f-48ed-bbea-9f1b27b911c8

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=ebec273b-0774-4ccd-90a9-c2a22144d623&var-namespace=ch-benchmark-pg-cdc-daily-20240207&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All&from=1707342785684&to=1707348581400

The symptom is same as the conclusion in #14943 (comment):

• Join amplification
• Many L0 files

which causes barrier piled up and backpressures the the source.

[lmatz]

which causes barrier piled up and backpressures the the source.

Remove it from blockers for now.

Join amplification
Many L0 files

Join amplification is expected as it is determined by the nature of the query and data
but wonder why many L0 files