(Metabase 0.50.5 is unavailable) +CrashOnOutOfMemoryError cpu problem about metabase

[paoliniluis]

We’re working on this as we speak

[paoliniluis]

ok we have some strings to pull from now

[paoliniluis]

@alperengunes if you don't send us what we need, we won't be able to identify the problem

[rob-pomelo]

also seeing significant performance regression since going from 0.49 -> 0.50, cpu usage spiking, will try get some logs

[perivamsi]

@alperengunes could you click on the "download diagnostics info" and send over that file to us after reviewing that there is no sensitive information there?

[LukeAbell]

Also seeing performance regression and CPU spikes.

[LukeAbell]

What can I share? It seems like it's mostly affecting large dashboards with a lot of questions. Looks like it's the metadata endpoint mostly.

Even loading the dashboard spikes CPU usage of our pods:

[matotias]

@matotias and what engine/sizing are you using as well

We went from 0.49.12 to 0.50.4 and the spikes started, then we updated again to 0.50.5 but we're still seeing them

We use postgres, 4 cpu 16 gigs of ram

[matotias]

Update on our case: we went back to v0.49.12 and the database load is normal again

[psneha716]

Hi,
I tried disabling query parsing using MB_SQL_PARSING_ENABLED=false but it's of no help. Metabase is still crashing with OOM.
Version: v0.50.1

Context:

Metabase deployed on ECS with EC2.
CPU per task: 3800 units (3.711 vCPU)
Memory per task: 3700 MiB (3.613 GB)

2 tasks deployed currently

[cheyuriy]

Similar issue with CPU spikes after upgrading from 0.49.x to 0.50.6.
Using Postgres 14 as a DB. Metabase is deployed on 2CPU-12GB in GCE and DB is in CloudSQL.

Upgrade was in the end of 06/24.

[LukeAbell]

@uladzimirdev I had to clear my local browser cache to resolve the issue. We have metabase behind a google cloud CDN but that didn't be the issue.

[noahmoss]

Hi @psneha716 — I don't see any errors in the downgrading logs that indicate specifically why it didn't work. But I did fix an issue related to downgrading in Metabase 50.7, the latest release. Could you try starting that version up, and then running the downgrade command with it?

If that doesn't work could you open a new issue with the full logs?

[uladzimirdev]

@alperengunes could you please add more details about your configuration?

[paoliniluis]

@alperengunes please post the logs, specs... at least you're not running Metabase correctly there since you're not probably tweaking the heap limit

[alperengunes]

In 49.x updates I don't have this problem, but in 50.x updates I have this problem. There are no different changes I have made.

[alperengunes]

not only this but I have been getting expression error since I updated metabase 50.x

[perivamsi]

@LukeAbell any diagnostics you can share with us that can help debug this?

[rob-pomelo]

also running in gke and have had similar cpu spike on upversion just like ^

[uladzimirdev]

@rob-pomelo https://www.metabase.com/docs/latest/troubleshooting-guide/diagnostic-info

[LukeAbell]

@perivamsi Just emailed support logs for the dashboard.

[paoliniluis]

@LukeAbell @rob-pomelo and @alperengunes can you add MB_SQL_PARSING_ENABLED=false, reboot and then let us know if this keeps happening?

@alperengunes
2024-06-18 14:33:00,039 WARN malli.fn :: Invalid output
ERROR middleware.add-source-metadata :: Error determining expected columns for query
clojure.lang.ExceptionInfo: Error preprocessing query in metabase.query_processor.preprocess
Caused by: clojure.lang.ExceptionInfo: Invalid query

please post the questions that spit those lines

[matotias]

Not sure if it's the same issue, but we started seeing huge spikes in the database after updating, these updates seem to be the problem

[paoliniluis]

@matotias what version did you upgrade from?

[paoliniluis]

@matotias and what engine/sizing are you using as well

[LukeAbell]

@paoliniluis Didn't seem to change much. I think it's permission or sandbox related because it's much faster as a super admin vs a normal user that's sandboxed using full app embed.

[LukeAbell]

Check out the difference in time between sandboxed (right) and super admin (left).

[paoliniluis]

@LukeAbell I'm trying to reproduce what you're seeing and I'm wondering how it's taking 4 seconds to get a 30kb response. Can you describe a little more the queries? how are queries built (based on questions or models?). Are tables big? many joins?

[paoliniluis]

@matotias and what engine/sizing are you using as well

We went from 0.49.12 to 0.50.4 and the spikes started, then we updated again to 0.50.5 but we're still seeing them

We use postgres, 4 cpu 16 gigs of ram

4 cpu and 16 gigs of ram is the postgres sizing right?

[LukeAbell]

@paoliniluis want to hop on a quick video chat?

[paoliniluis]

@LukeAbell give me a couple of hours to try to go to the call with some answers, can you make it tomorrow first time in the morning?

[LukeAbell]

@paoliniluis Tomorrow at 10am EST work?

[LukeAbell]

@LukeAbell I'm trying to reproduce what you're seeing and I'm wondering how it's taking 4 seconds to get a 30kb response. Can you describe a little more the queries? how are queries built (based on questions or models?). Are tables big? many joins?

Queries are based on questions (no models involved). Not a ton of joins. Using bigquery- table has about 800k rows.

Only 6 tables are returned in that query_metadata call.

[paoliniluis]

Thanks @LukeAbell, do you have observability in the app db to see if there's something weird there?

[LukeAbell]

Yep- doesn't seem like anything weird there. Less than 20% utilization. Running MySQL.

[matotias]

@matotias and what engine/sizing are you using as well

We went from 0.49.12 to 0.50.4 and the spikes started, then we updated again to 0.50.5 but we're still seeing them
We use postgres, 4 cpu 16 gigs of ram

4 cpu and 16 gigs of ram is the postgres sizing right?

Yes!

[alperengunes]

@alperengunes could you click on the "download diagnostics info" and send over that file to us after reviewing that there is no sensitive information there?

checking

[alperengunes]

@LukeAbell @rob-pomelo and @alperengunes can you add MB_SQL_PARSING_ENABLED=false, reboot and then let us know if this keeps happening?

@alperengunes 2024-06-18 14:33:00,039 WARN malli.fn :: Invalid output ERROR middleware.add-source-metadata :: Error determining expected columns for query clojure.lang.ExceptionInfo: Error preprocessing query in metabase.query_processor.preprocess Caused by: clojure.lang.ExceptionInfo: Invalid query

please post the questions that spit those lines

not working :'( same

[paoliniluis]

@matotias is your app db a postgres or mysql?

[matotias]

postgres

[naveenthontepu]

Metabase is crashing every 3 to 4 hours in our case also. There is no significant jump in the CPU utilisation but the RAM utilisation has significantly increased and is consuming the complete 4 gb ram we have on the metabase server.

Looking at the tasks running on our server by running "htop" we are seeing that there are too many tasks metabase is keeping active even when there is nobody using metabase.

[perivamsi]

@naveenthontepu could you send us the output of htop and logs of the tasks that Metabase is keeping active?

[paoliniluis]

Also: any specific environment variable that you’re using?

[naveenthontepu]

There are no environment variables that we are using.

the command we are using to run metabase specifically mentions to use only 2gb of the 4gb ram but it is not being considered and using it 4gb and crashing. In the above screenshot also we can see that metabase is consuming 71% memory which is ~3gb.

[noahmoss]

@LukeAbell what version of MySQL are you on?

[LukeAbell]

@noahmoss 8.0.31

[perivamsi]

@LukeAbell we have some changes coming that that might improve the performance. would you be able to upgrade to 50.7 and test? which version are you on right now? is it easy for you to downgrade if 50.7 is still slow for you?

[LukeAbell]

@perivamsi yep just let me know when there's a tagged release. Or do I need to test on a custom branch?

[perivamsi]

no need to test a custom branch, I'll give you a proper release link once ready

[shrey-locad]

exactly same problem since the upgrade to 0.50. memory and cpu spiking intermittently bringing down the whole container every 3-4 hours. We will give it 1-2 more days before downgrading back to 0.49

[perivamsi]

@shrey-locad sorry about that! are you open to having a call that can help us debug this better? as I said, we are working on this but it would be helpful to have additional data points around the memory and CPU spikes. also, hi from another IITB alum!

[shrey-locad]

@perivamsi - haha, that's a pleasant surprise. Hi 👋🏽

It might be hard to debug over a call, especially since the crashes are unpredictable and intermittent. We have been unable to identify a specific trigger either. The latest crash was in the middle of the night where we're pretty sure none of our users were online. How about we gave you ssh access to the instance instead? Call works too. Either way. [email protected]

[perivamsi]

That works, we just need to look at the logs. I'll email you.

[perivamsi]

@LukeAbell @shrey-locad @cheyuriy can you all please upgrade to 50.7 which has some fixes for the performance issues?

https://github.com/metabase/metabase/releases/tag/v0.50.7

[LukeAbell]

@perivamsi I upgraded but had to quickly downgrade because it completely broke the application. Downgrading back to v1.50.6 fixed the issue. I tried clearing our CDN cache and flushing browser cache but it didn't help.

Assets wouldn't load giving an error:

Refused to execute script from 'HOST/app/dist/runtime.ba18610064ff9fba6745.js' because its MIME type ('') is not executable, and strict MIME type checking is enabled.

[perivamsi]

We're on it

[bshepherdson]

That sounds like #39449. If force-refresh doesn't work, try another browser?

It does seem like it was a transitory issue in the past.

[LukeAbell]

@perivamsi @bshepherdson Completely cleared cache and that worked. Unfortunately not much of an improvement in performance (if any) 😔

• Query metadata taking 4-6 seconds sandboxed
• Queries taking 3-5 seconds sandboxed
• CPU usage still spiking like crazy loading a single dashboard

[uladzimirdev]

@LukeAbell just for the record, what kind of cache have you cleared? what CDN do you use? we're struggling to reproduce the issue, so any info would be helpful (some users had it during upgrades from older versions)

[psneha716]

Hi folks,
Could someone who has already downgraded their metabase version, help me out with the downgrading process? Tried downgrading, but it isn't working. Any help would be greatly appreciated. Thanks!

Cluster configuration:

ECS cluster with 2 tasks.
Task CPU: 3800 units (3.711 vCPU)
Task memory: 3700 MiB (3.613 GB)

Steps that I followed for downgrading (followed instructions from here):

• Stopped the container running v0.50.1 metabase instance

• Ran the following on the EC2 instance directly:

$docker run --rm metabase/metabase:v0.50.1 "migrate down"
Warning: environ value jdk-11.0.23+9 for key :java-version has been overwritten with 11.0.23
2024-06-12 21:08:25,427 INFO metabase.util :: Maximum memory available to JVM: 1.9 GB
2024-06-12 21:08:27,762 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
2024-06-12 21:08:28,566 WARN db.env :: WARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, see https://metabase.com/docs/latest/operations-guide/migrating-from-h2.html
2024-06-12 21:08:32,573 INFO driver.impl :: Registered abstract driver :sql  🚚
2024-06-12 21:08:32,579 INFO driver.impl :: Registered abstract driver :sql-jdbc (parents: [:sql]) 🚚
2024-06-12 21:08:32,585 INFO metabase.util :: Load driver :sql-jdbc took 32.7 ms
2024-06-12 21:08:32,586 INFO driver.impl :: Registered driver :h2 (parents: [:sql-jdbc]) 🚚
2024-06-12 21:08:32,717 INFO driver.impl :: Registered driver :mysql (parents: [:sql-jdbc]) 🚚
2024-06-12 21:08:32,744 INFO driver.impl :: Registered driver :postgres (parents: [:sql-jdbc]) 🚚
2024-06-12 21:08:34,401 INFO metabase.core ::
Metabase v0.50.1 (cc4ca82)

Copyright © 2024 Metabase, Inc.

Metabase Enterprise Edition extensions are NOT PRESENT.
2024-06-12 21:08:34,657 INFO db.setup :: Setting up Liquibase...
2024-06-12 21:08:35,245 INFO db.liquibase :: Updating liquibase table to reflect consolidated changeset filenames
2024-06-12 21:08:35,254 INFO db.liquibase :: No migration lock found.
2024-06-12 21:08:35,255 INFO db.liquibase :: Migration lock acquired.
2024-06-12 21:08:35,259 INFO db.setup :: Liquibase is ready.
2024-06-12 21:08:35,262 INFO db.liquibase :: No migration lock found.
2024-06-12 21:08:35,263 INFO db.liquibase :: Migration lock acquired.
2024-06-12 21:08:35,264 INFO db.liquibase :: Rolling back app database schema to version 49

• Tried spinning up a container using v0.49.15, but getting the following error:

2024-06-12 21:10:56,380 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
--
2024-06-12 21:10:56,382 INFO db.setup :: �[36mVerifying postgres Database Connection ...�[0m
2024-06-12 21:10:56,724 INFO db.setup :: Successfully verified PostgreSQL 13.13 application database connection. ✅
2024-06-12 21:10:56,725 INFO db.setup :: �[36mChecking if a database downgrade is required...�[0m
2024-06-12 21:10:56,834 ERROR middleware.log :: �[31mGET /api/health 503 287.2 µs (0 DB calls)
{:status "initializing", :progress 0.3}
�[0m
2024-06-12 21:10:56,844 ERROR middleware.log :: �[31mGET /api/health 503 182.0 µs (0 DB calls)
{:status "initializing", :progress 0.3}
�[0m
2024-06-12 21:10:57,344 ERROR metabase.core :: Metabase Initialization FAILED
clojure.lang.ExceptionInfo: �[31mERROR: Downgrade detected.�[0m
Your metabase instance appears to have been downgraded without a corresponding database downgrade.
You must run `java -jar metabase.jar migrate down` from version 50.
Once your database has been downgraded, try running the application again.

[psneha716]

Hi @noahmoss ,
It did not work. Created a new issue here.