Quarkus hibernate&oracle on update schema hangs up without any meaningful messages about quarkus

[quarkus-bot]

/cc @geoand (kotlin), @gsmet (hibernate-orm), @yrodiere (hibernate-orm)

[yrodiere]

@gituwalie Thanks for reporting.

I'm away this week so I can't have a look right now, but it seems your reproducer is relying on external databases, with no instructions as to how to set them up.

Does your problem happen also when using dev services?

If not, can you provide information on how to set up your Oracle DB? That might give us hints as to what specific Oracle DB behavior triggers the problem. From there @beikov may have an idea...

[beikov]

Please start your app in debug mode and when you feel like it hangs, suspend the VM with the debug controls. Then post the stack trace of the thread that seems hanging.

[gituwalie]

@yrodiere I don't use devservice. Also setup of database can't be shared because of company policies. However I can tell that there are around 3500 tables there. Almost same amount of view(3700) Both of them may be quite big (up to 140 columns).

@beikov I've extracted threaddumps .
1.txt
2.txt
Also upon debug I've checked that
org.hibernate.tool.schema.extract.internal.AbstractInformationExtractorImpl#populateTablesWithColumns
Scans all the tables&views like one by one. Can it be an issue? I've never drilled into hibernate insides but I don't think it's ok to scan all tables for validation.

Seems I'm not the only one with such a problem.
#34674

[beikov]

AFAIU, Hibernate is in the process of reading the metadata about your tables but is constantly waiting for data, which I assume since the CPU time didn't change a lot, but the elapsed time does.

This tells me that your database is potentially waiting for a lock. Do you have processes running that require exclusive locks on the table metadata? I'm no Oracle expert, but to me, this seems like a problem of your environment.

[gituwalie]

Well, after hours of checks, it just reads all the meta from tables in specified schema. Reads it very slowly. So I've created a new separate schema where only 3 new tables appeared and it worked fine. I guess it's not an option to use "update" hibernate option for production like schemas. Issue can be closed.

[beikov]

You shouldn't use hbm2ddl update in production anyway for other reasons, but this is still interesting since tables are also read when using validate.
We'd very much appreciate if you could check what is happening on the database when this process takes so long, because it shouldn't. We just call the java.sql.DatabaseMetaData#getTables method, which is a method on the JDBC driver.

[yrodiere]

FWIW, this relates to #34674, which is apparently a very common problem, but one that's hard to investigate because almost nobody with such a big schema can share it because of (understandable) company policies.
It's currently under investigation using a schema that someone was able to share, but the more information we get, the better. Especially since this might actually be multiple problems, some of them db-specific. So, share you schemas, people! At least whoever can :)

cc @Sanne yet another one... we'll need an update on current investigations :)

[Sanne]

cc @Sanne yet another one... we'll need an update on current investigations :)

To be fair what I was looking into this week is not related to #34674 , and I wasn't planning to look into that.

I've been looking into hot-reload issues with very large models, like https://github.com/topicusonderwijs/tribe-krd-quarkus . This particular model is stressing the efficiency of Quarkus's bytecode enhancement and processing the (Java) model, the DB doesn't come into play.

I've had some success in improving this (various patches coming soon, some in Hibernate ORM, some in Quarkus, but don't hold your breath as they don't seem related to this other issue.

---

Incidentally I see the concerning word "hang" being mentioned above. Is it just very slow or is there some deadlock in play? (Not able to investigate myself but suggesting to check that - one might break out of deadlocks as most locks do timeout eventually, but that would certainly explain extreme slowness)

[Sanne]

I guess it's not an option to use "update" hibernate option for production like schemas. Issue can be closed.

For sure I would agree we don't want to make "update" a feature recommended for "production" - or even ever suggest this - but if there's a notorious inefficiency we can look into it.

But please track the issue on the Hibernate issue tracker

• https://hibernate.atlassian.net/

and suggest some way for us to reproduce - perhaps you could adapt the model at https://github.com/topicusonderwijs/tribe-krd-quarkus to trigger this?