Can't load schema of database in Spark SQL about metabase

[tsmacdonald]

Hi @lakechain , do you see anything about the Database Sync for this DB happening in /admin/troubleshooting/tasks ? It'd be helpful to know if the issue is the sync not running, the sync failing, or the sync allegedly succeeding but not populating things correctly.

[calherries]

@lakechain I tested syncing the test database we have for Spark on 0.50.8, and the tables are syncing fine, so this is likely something specific about your setup. Can you share the log messages during database sync?

1. go to the Databases page for that database in Admin Settings and click "Sync database schema now"
2. go to Admin Settings > Troubleshooting > Logs. The database logs for syncing database schema should start with something like this:

2024-07-03 18:04:53,996 INFO sync.util :: STARTING: Sync sparksql Database 1 ''test-data''
2024-07-03 18:04:53,997 INFO sync.util :: STARTING: Sync metadata for sparksql Database 1 ''test-data''
...

[lakechain]

@calherries @tsmacdonald table sync is fine, but schemas are not. And I don't understand why Database name is required, in Spark SQL, Schema and Database are one, so when I enter database name as "default", no tables appear in the Database browser of metabase, but when I enter the name of a schema in the database name input, the tables of that schema appear in the Database browser.
How can I display all schemas (databases) in my warehouse on the database browser of Metabase?

[calherries]

Ah yes, now I understand your issue. Having to specify a single schema is indeed a limitation of the Spark driver. Can you confirm whether this issue represents the solution you'd like?

Specifically:

if you leave the database blank, the UI will show the metadata from all databases.

[lakechain]

Maybe that's what I need.
But I read in the SparkSQL connector documentation. I see that the database name is not required. https://www.metabase.com/docs/latest/databases/connections/sparksql

[lakechain]

@calherries I can't leave the database blank, it's required, Is this a bug?

[calherries]

I read in the SparkSQL connector documentation. I see that the database name is not required.

Can you point out exactly where you see that in the documentation?

I think this is definitely a limitation in the design of the Spark SQL driver. But it looks like it was intentionally designed this way, so I'm not sure it's a bug. As you said before:

when I enter the name of a schema in the database name input, the tables of that schema appear in the Database browser.

Is it a feasible workaround to create a database in Metabase for each database/schema in your Spark SQL database?

[lakechain]

In the document I don't see anything that says I need to input a database name. I'll try some ways to see if I can get around it, thanks for your help.

[calherries]

Thanks, please update this issue if you find the workaround satisfactory. Given you and a few others are obviously having issues with this, I will see if there's an easy way to fix this limitation. No promises though! There might be a good reason for this

[darksciencebase]

@lakechain did the workaround work for you?

[lakechain]

@darksciencebase I am having to create a database for each schema in the data warehouse. This is probably the best way possible.