nightly-20240130 nexmark-q5-many-windows perf degradation about risingwave

[lmatz]

The throughput increased likely because of 0cd9ff1 #14558

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240115

now drops because of 9417409 #14855

14588 vs 14855, what a coincidence

[TennyZhuang]

Interesting, but I can't explain this phenomenon. IMO this PR should only have an effect on UDF :)

[fuyufjh]

Interesting +1, Worth investigating the cause, I think

[cyliu0]

Any update? The perf never came back to the original level http://metabase.risingwave-cloud.xyz/question/1304-nexmark-q5-many-windows-blackhole-medium-1cn-avg-source-output-rows-per-second-rows-s-history-thtb-266?start_date=2024-01-22

[fuyufjh]

@TennyZhuang Any updates?

[fuyufjh]

CREATE SINK nexmark_q5_many_windows
AS
SELECT
    AuctionBids.auction, AuctionBids.num
FROM (
    SELECT
        bid.auction,
        count(*) AS num,
        window_start AS starttime
    FROM
        HOP(bid, date_time, INTERVAL '5' SECOND, INTERVAL '5' MINUTE)
    GROUP BY
        bid.auction,
        window_start
) AS AuctionBids
JOIN (
	SELECT
        max(CountBids.num) AS maxn,
        CountBids.starttime_c
	FROM (
		SELECT
            count(*) AS num,
            window_start AS starttime_c
		FROM
            HOP(bid, date_time, INTERVAL '5' SECOND, INTERVAL '5' MINUTE)
        GROUP BY
            bid.auction,
            window_start
		) AS CountBids
	GROUP BY
        CountBids.starttime_c
	) AS MaxBids
ON
    AuctionBids.starttime = MaxBids.starttime_c AND
    AuctionBids.num >= MaxBids.maxn
WITH ( connector = 'blackhole', type = 'append-only', force_append_only = 'true');

Plan:

 StreamSink { type: append-only, columns: [auction, num, window_start(hidden), window_start#1(hidden)] }
 └─StreamProject { exprs: [$expr1, count, window_start, window_start] }
   └─StreamFilter { predicate: (count >= max(count)) }
     └─StreamHashJoin { type: Inner, predicate: window_start = window_start }
       ├─StreamExchange { dist: HashShard(window_start) }
       │ └─StreamShare { id: 7 }
       │   └─StreamHashAgg [append_only] { group_key: [$expr1, window_start], aggs: [count] }
       │     └─StreamExchange { dist: HashShard($expr1, window_start) }
       │       └─StreamHopWindow { time_col: $expr2, slide: 00:00:05, size: 00:05:00, output: [$expr1, window_start, _row_id] }
       │         └─StreamProject { exprs: [Field(bid, 0:Int32) as $expr1, Field(bid, 5:Int32) as $expr2, _row_id] }
       │           └─StreamFilter { predicate: IsNotNull(Field(bid, 5:Int32)) AND (event_type = 2:Int32) }
       │             └─StreamRowIdGen { row_id_index: 4 }
       │               └─StreamSource { source: nexmark, columns: [event_type, person, auction, bid, _row_id] }
       └─StreamProject { exprs: [window_start, max(count)] }
         └─StreamHashAgg { group_key: [window_start], aggs: [max(count), count] }
           └─StreamExchange { dist: HashShard(window_start) }
             └─StreamShare { id: 7 }
               └─StreamHashAgg [append_only] { group_key: [$expr1, window_start], aggs: [count] }
                 └─StreamExchange { dist: HashShard($expr1, window_start) }
                   └─StreamHopWindow { time_col: $expr2, slide: 00:00:05, size: 00:05:00, output: [$expr1, window_start, _row_id] }
                     └─StreamProject { exprs: [Field(bid, 0:Int32) as $expr1, Field(bid, 5:Int32) as $expr2, _row_id] }
                       └─StreamFilter { predicate: IsNotNull(Field(bid, 5:Int32)) AND (event_type = 2:Int32) }
                         └─StreamRowIdGen { row_id_index: 4 }
                           └─StreamSource { source: nexmark, columns: [event_type, person, auction, bid, _row_id] }

[fuyufjh]

It's indeed caused by #14558, but the reason is unknown. Will continue to investigate.

CPU flamegraph: profile results.zip