Comments (11)
While not an explicit restriction, we only test the dialect on warehouses / clusters with unity catalog enabled. Can you share which line of the example script raises this exception?
from databricks-sql-python.
Sorry for the delay, I re-ran that sample again and interestingly get a different error. Here's the full error and stack trace
base = declarative_base(bind=engine)
Traceback (most recent call last):
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/client.py", line 472, in execute
execute_response = self.thrift_backend.execute_command(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 833, in execute_command
return self._handle_execute_response(resp, cursor)
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 925, in _handle_execute_response
final_operation_state = self._wait_until_command_done(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 767, in _wait_until_command_done
self._check_command_not_in_error_or_closed_state(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 484, in _check_command_not_in_error_or_closed_state
raise ServerOperationError(
databricks.sql.exc.ServerOperationError: Table constraints are only supported in Unity Catalog.
from databricks-sql-python.
AH I see why, The first error was run on the following
Catalog: samples
Schema: default
The most recent error was run on:
Catalog: hive_metastore
Schema: default
from databricks-sql-python.
Here's the trace for the initial error message
base = declarative_base(bind=engine)
Traceback (most recent call last):
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/client.py", line 472, in execute
execute_response = self.thrift_backend.execute_command(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 833, in execute_command
return self._handle_execute_response(resp, cursor)
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 925, in _handle_execute_response
final_operation_state = self._wait_until_command_done(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 767, in _wait_until_command_done
self._check_command_not_in_error_or_closed_state(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 484, in _check_command_not_in_error_or_closed_state
raise ServerOperationError(
databricks.sql.exc.ServerOperationError: [UC_COMMAND_NOT_SUPPORTED] Create sample tables/views is not supported in Unity Catalog.
from databricks-sql-python.
Okay I don't see a bug here. There are two issues at play.
-
You can't create tables and views on the
samples
catalog. This catalog is there for demonstration purposes (read about it in the Databricks docs here) and is effectively read-only. -
The error with
hive_metastore
is because that catalog is special -- it maintains backward compatibility with catalogs created before Unity Catalog was available. So although thehive_metastore
catalog is included in the Unity Catalog, it doesn't have any UC features enabled. It's there purely for backward compatibility.
To move through this you need to just create a new catalog in your UC and point SQLAlchemy at it. Many organisations simply call this catalog main
.
from databricks-sql-python.
Thanks @susodapop After further review we don't have a UC created yet but are working on that now. It seems like a UC is required to use the databricks-sql-connector. Is that accurate?
from databricks-sql-python.
No UC is not required to use databricks-sql-connector.
The implicit restriction I mentioned above only applies to the SQLAlchemy dialect.
from databricks-sql-python.
Thanks @susodapop. Interesting, I'm getting this error: [UC_NOT_ENABLED] Unity Catalog is not enabled on this cluster. When trying to create a new catalog/schema with the sql connector.
create_catalog_stmt = f"CREATE CATALOG IF NOT EXISTS {catalog_name}"
create_schema_stmt = f"CREATE SCHEMA IF NOT EXISTS {catalog_name}.{schema_name}"
with sql.connect(
server_hostname=server_hostname,
http_path=http_path,
access_token=access_token,
) as connection:
with connection.cursor() as cursor:
# Create Catalog
cursor.execute(create_catalog_stmt)
logger.info("Created Catalog %r", catalog_name)
# Create Schema
cursor.execute(create_schema_stmt)
logger.info("Created Schema %r in Catalog %r", catalog_name, schema_name)
Trace:
Traceback (most recent call last):
File "scripts/create_catalog_schema.py", line 39, in
create_catalog_schema("rbales_test_catalog", "rbales_test_schema")
File "scripts/create_catalog_schema.py", line 28, in create_catalog_schema
cursor.execute(create_catalog_stmt)
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/client.py", line 472, in execute
execute_response = self.thrift_backend.execute_command(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 833, in execute_command
return self._handle_execute_response(resp, cursor)
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 925, in _handle_execute_response
final_operation_state = self._wait_until_command_done(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 767, in _wait_until_command_done
self._check_command_not_in_error_or_closed_state(
File "/Users/rbales/Library/Caches/pypoetry/virtualenvs/active-ai-api-NyKgQ79e-py3.8/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 484, in _check_command_not_in_error_or_closed_state
raise ServerOperationError(
databricks.sql.exc.ServerOperationError: [UC_NOT_ENABLED] Unity Catalog is not enabled on this cluster.
from databricks-sql-python.
That error comes directly from the cluster (i.e. it's not an issue with the python connector). Have you confirmed that UC is enabled for this cluster?
from databricks-sql-python.
It's not, no UC is enabled.
from databricks-sql-python.
Hi @ryanbales! Were you able to enable UC for your cluster and check if your issue reproduces there?
from databricks-sql-python.
Related Issues (20)
- Proxy authentication not working
- Is it possible to insert an arrow table? HOT 2
- Insert performance is woeful! 😢 HOT 11
- cursor.execute returing error 'NoneType' object is not iterable HOT 15
- [Document] Support databricks-cli authentication HOT 1
- Dash/Minus in column name causes UNBOUND_SQL_PARAMETER in bind values HOT 2
- Extremely slow import times on Python 3.12 HOT 9
- [sqlalchemy] execute("select 1") gives TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType' HOT 8
- `databricks.sql.connect` hangs in a long retrying loop when an invalid access token is used HOT 3
- Idea: arrow_record_batches cursor method
- SQLAlchemy engine from workspace level service principle? HOT 2
- Unable to write list/array type data HOT 2
- Issue with version 3.1.1
- Failure on cursor.fetchall() HOT 2
- Fixing a couple type problems. (adding py.typed, typing connect, returning Any from fetchall (which I failed to fix!)) HOT 2
- Connector reads 0 rows although Cluster returned results HOT 18
- support new Cursor attribute that provides information on completed commands HOT 1
- loosen, update, or widen pyarrow dependencies HOT 2
- ImportError: cannot import name 'sql' from partially initialized module 'databricks' HOT 2
- Unpin Thrift
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from databricks-sql-python.