[SUPPORT]Performance degrade for migrating from Hudi 0.7 to Hudi 0.14 about hudi

[KnightChess]

hoodie.simple.index.parallelism can be modified to adjust the parallelism of stage 121, but it may cause the parallelism of stage 120 to decrease. You can give it a try.

[bibhu107]

Hi @KnightChess Thanks for commenting. But my major doubt is why shuffle write is nearly doubled in hudi 0.14? And that is leading to issues in step 121

[KnightChess]

@bibhu107 if next parallelism is too bigger, shuffle data will grow. And the mayjor reason is task is too much because high, spark need to scheduler too mush task.

[KnightChess]

@bibhu107 and why shuffle data grow, I haven't looked at the code in detail; the following is just my guess. you have too much reducer, so the shuffle data may be need more meta. And on the other head, 0.7 -> 0.14, may be the shuffle's java object attr has change, this alse can cause diff. But I think parallelism is the major problem.

[KnightChess]

@bibhu107 does it can work for you? I misread the stack trace, so this parameter hasn't taken effect in the question stage. The parameter that needs to be set is a different one. If you are insert, try to set hoodie.insert.shuffle.parallelism, if upsert, set hoodie.upsert.shuffle.parallelism

[soumilshah1995]

Slack Thread https://apache-hudi.slack.com/archives/C4D716NPQ/p1718122404452279