[SUPPORT] Schema file too large and keeps growing, OOM when http handle it about hudi

[ad1happy2go]

@lei-su-awx Yes you are correct. the schema file grows with every commit and grows exponential which will cause this kind of issues. We need to redesign this schema commit file.

Created tracking JIRA for the same - https://issues.apache.org/jira/browse/HUDI-7481

[lei-su-awx]

@ad1happy2go got it, thanks for your reply.

[lei-su-awx]

Hi @ad1happy2go I was wondering if there is any configuration to compact schemacommit file or only store the latest schema or any other workaround?

[xiarixiaoyao]

@lei-su-awx
how many version of internalSchema stored in the schemacommit.
Only when there has been a schema change will a new version of the schema be generated. Do you have frequent schema changes for this job?

Normally, a table should not have too frequent schema changes.
If there is no change in field type, you can directly clear the. schema directory

[ad1happy2go]

@xiarixiaoyao Even without the schema changes, this grows infinitely, may be as sync time always change with every update.

[xiarixiaoyao]

@ad1happy2go @lei-su-awx will take a look for hudi 0.14.0.
we use hudi 0.11.0 and have no such problem.