perf: improve nexmark q102 scaling up performance about risingwave

[lmatz]

Some notable phenomena:

1. Join Executor Barrie Align is much higher when scaling up

4X:

1X:

2. HashJoin executor's throughput is much worse when scaling up

4X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1706396338000&orgId=1&to=1706398141000&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=nexmark-lt-4x-1cn-affinity-weekly-20240127&editPanel=63

1X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1706413292000&to=1706415095000&var-namespace=nexmark-1cn-affinity-weekly-20240127&editPanel=63

3. StatelessSimpleAgg executor's throughput is much worse when scaling up

4X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1706396338000&orgId=1&to=1706398141000&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=nexmark-lt-4x-1cn-affinity-weekly-20240127&editPanel=63

1X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1706413292000&to=1706415095000&var-namespace=nexmark-1cn-affinity-weekly-20240127&editPanel=63

No wonder, SimpleAgg is bad too since it is on the other side of exchange after StatelessSimpleAgg.
4X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1706396338000&orgId=1&to=1706398141000&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=nexmark-lt-4x-1cn-affinity-weekly-20240127&editPanel=63

1X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1706413292000&to=1706415095000&var-namespace=nexmark-1cn-affinity-weekly-20240127&editPanel=63

4. HashAgg executor throughput is worse when scaling up

There are two HashAgg in the query plan, but since they are both worse, it does not matter which one is which one (How to differentiate two though?)

4X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1706396338000&orgId=1&to=1706398141000&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=nexmark-lt-4x-1cn-affinity-weekly-20240127&editPanel=63

1X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1706413292000&to=1706415095000&var-namespace=nexmark-1cn-affinity-weekly-20240127&editPanel=63

5. Dynamic executor throughput is better when scaling up

4X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1706396338000&orgId=1&to=1706398141000&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=nexmark-lt-4x-1cn-affinity-weekly-20240127&editPanel=63

1X:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1706413292000&to=1706415095000&var-namespace=nexmark-1cn-affinity-weekly-20240127&editPanel=63

[lmatz]

Since:

1. The bottom HashAgg is the first stateful operator in the branch of subplan, it does not make sense that its performance will decrease when scaling up. probably due to back-pressure
2. StatelessSimpleAgg and SimpleAgg are both too simple to be the root cause. (may still take a look) probably also due to back-pressure
3. Dynamic filter executor's throughput gets better when scaling up

I suppose HashJoin and Exchange may need to be investigated first?