Test failure: `test_migration_job_ext_hms[regular]`,about databrickslabs/ucx

Comments (2)

github-actions commented on June 3, 2024

❌ test_migration_job_ext_hms[regular]: TimeoutError: timed out after 0:20:00: (37m23.392s)

TimeoutError: timed out after 0:20:00:
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/zLEr
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.migrate_qpc8t
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qpc8t: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qpc8t', metastore_id=None, name='migrate_qpc8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_tt63y (id INT, value STRING)
07:19 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_tt63y: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_tt63y
07:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_tt63y', metastore_id=None, name='ucx_tt63y', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qpc8t/ucx_tt63y', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_t4pzy USING json location 'dbfs:/tmp/ucx_test_Ear2... (74 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_t4pzy: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_t4pzy
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_t4pzy', metastore_id=None, name='ucx_t4pzy', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_Ear2', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_t1tgx USING CSV OPTIONS (header=true) LOCATION 'db... (24 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_t1tgx: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_t1tgx
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_t1tgx', metastore_id=None, name='ucx_t1tgx', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/zLEr', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_qpc8t.ucx_thzzk AS SELECT * FROM hive_metastore.migrate_qpc8t... (10 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_thzzk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_thzzk
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_thzzk', metastore_id=None, name='ucx_thzzk', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_tt63y', view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_qpc8t.ucx_ttqro AS SELECT * FROM hive_metastore.migrate_qpc8t... (10 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_ttqro: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_ttqro
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_ttqro', metastore_id=None, name='ucx_ttqro', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_thzzk', view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE CATALOG ucx_cyajz
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1715239227924, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cyajz', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cyajz', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1715239227924, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA ucx_cyajz.migrate_qpc8t
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cyajz.migrate_qpc8t: https://DATABRICKS_HOST/explore/data/ucx_cyajz/migrate_qpc8t
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cyajz', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cyajz.migrate_qpc8t', metastore_id=None, name='migrate_qpc8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sv5l5
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sv5l5: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sv5l5
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sv5l5', metastore_id=None, name='ucx_sv5l5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.tables (catalog STRING NOT NULL, database STRING NOT NULL, ... (207 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.tables (catalog, database, name, object_type, table_format, location, view... (736 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.mounts (name STRING NOT NULL, source STRING NOT NULL) USING... (6 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.mounts (name, source) VALUES ('/mnt/TEST_MOUNT_NAME/a', 'abfss://TEST_MOUNT_NAME@labsCLOUD_ENVth... (29 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.groups (id_in_workspace STRING NOT NULL, name_in_workspace ... (164 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.groups (id_in_workspace, name_in_workspace, name_in_account, temporary_nam... (130 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.grants (principal STRING NOT NULL, action_type STRING NOT N... (152 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.grants (principal, action_type, catalog, database, table, view, udf, any_f... (118 more bytes)
[gw5] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/zLEr
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.migrate_qpc8t
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qpc8t: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qpc8t', metastore_id=None, name='migrate_qpc8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_tt63y (id INT, value STRING)
07:19 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_tt63y: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_tt63y
07:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_tt63y', metastore_id=None, name='ucx_tt63y', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qpc8t/ucx_tt63y', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_t4pzy USING json location 'dbfs:/tmp/ucx_test_Ear2... (74 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_t4pzy: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_t4pzy
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_t4pzy', metastore_id=None, name='ucx_t4pzy', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_Ear2', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_t1tgx USING CSV OPTIONS (header=true) LOCATION 'db... (24 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_t1tgx: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_t1tgx
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_t1tgx', metastore_id=None, name='ucx_t1tgx', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/zLEr', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_qpc8t.ucx_thzzk AS SELECT * FROM hive_metastore.migrate_qpc8t... (10 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_thzzk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_thzzk
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_thzzk', metastore_id=None, name='ucx_thzzk', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_tt63y', view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_qpc8t.ucx_ttqro AS SELECT * FROM hive_metastore.migrate_qpc8t... (10 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_ttqro: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_ttqro
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_ttqro', metastore_id=None, name='ucx_ttqro', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_thzzk', view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE CATALOG ucx_cyajz
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1715239227924, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cyajz', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cyajz', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1715239227924, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA ucx_cyajz.migrate_qpc8t
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cyajz.migrate_qpc8t: https://DATABRICKS_HOST/explore/data/ucx_cyajz/migrate_qpc8t
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cyajz', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cyajz.migrate_qpc8t', metastore_id=None, name='migrate_qpc8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sv5l5
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sv5l5: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sv5l5
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sv5l5', metastore_id=None, name='ucx_sv5l5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.tables (catalog STRING NOT NULL, database STRING NOT NULL, ... (207 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.tables (catalog, database, name, object_type, table_format, location, view... (736 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.mounts (name STRING NOT NULL, source STRING NOT NULL) USING... (6 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.mounts (name, source) VALUES ('/mnt/TEST_MOUNT_NAME/a', 'abfss://TEST_MOUNT_NAME@labsCLOUD_ENVth... (29 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.groups (id_in_workspace STRING NOT NULL, name_in_workspace ... (164 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.groups (id_in_workspace, name_in_workspace, name_in_account, temporary_nam... (130 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.grants (principal STRING NOT NULL, action_type STRING NOT N... (152 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.grants (principal, action_type, catalog, database, table, view, udf, any_f... (118 more bytes)
07:21 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.tTKC/config.yml) doesn't exist.
07:21 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:21 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:21 INFO [databricks.labs.ucx.install] Fetching installations...
07:21 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.tTKC is corrupted. Skipping...
07:21 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:21 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:21 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:21 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:21 INFO [databricks.labs.ucx.install] Installing UCX v0.23.1+220240509072125
07:21 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sv5l5
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.external_locations (location STRING NOT NULL... (40 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:22 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.tTKC/README for the next steps.
07:22 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/682900195538395
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/zLEr
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.migrate_qpc8t
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qpc8t: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qpc8t', metastore_id=None, name='migrate_qpc8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_tt63y (id INT, value STRING)
07:19 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_tt63y: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_tt63y
07:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_tt63y', metastore_id=None, name='ucx_tt63y', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qpc8t/ucx_tt63y', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_t4pzy USING json location 'dbfs:/tmp/ucx_test_Ear2... (74 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_t4pzy: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_t4pzy
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_t4pzy', metastore_id=None, name='ucx_t4pzy', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_Ear2', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_qpc8t.ucx_t1tgx USING CSV OPTIONS (header=true) LOCATION 'db... (24 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_t1tgx: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_t1tgx
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_t1tgx', metastore_id=None, name='ucx_t1tgx', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/zLEr', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_qpc8t.ucx_thzzk AS SELECT * FROM hive_metastore.migrate_qpc8t... (10 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_thzzk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_thzzk
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_thzzk', metastore_id=None, name='ucx_thzzk', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_tt63y', view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_qpc8t.ucx_ttqro AS SELECT * FROM hive_metastore.migrate_qpc8t... (10 more bytes)
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qpc8t.ucx_ttqro: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qpc8t/ucx_ttqro
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_ttqro', metastore_id=None, name='ucx_ttqro', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_thzzk', view_dependencies=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE CATALOG ucx_cyajz
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1715239227924, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cyajz', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cyajz', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1715239227924, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA ucx_cyajz.migrate_qpc8t
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cyajz.migrate_qpc8t: https://DATABRICKS_HOST/explore/data/ucx_cyajz/migrate_qpc8t
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cyajz', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cyajz.migrate_qpc8t', metastore_id=None, name='migrate_qpc8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sv5l5
07:20 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sv5l5: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sv5l5
07:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sv5l5', metastore_id=None, name='ucx_sv5l5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.tables (catalog STRING NOT NULL, database STRING NOT NULL, ... (207 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.tables (catalog, database, name, object_type, table_format, location, view... (736 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.mounts (name STRING NOT NULL, source STRING NOT NULL) USING... (6 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.mounts (name, source) VALUES ('/mnt/TEST_MOUNT_NAME/a', 'abfss://TEST_MOUNT_NAME@labsCLOUD_ENVth... (29 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.groups (id_in_workspace STRING NOT NULL, name_in_workspace ... (164 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.groups (id_in_workspace, name_in_workspace, name_in_account, temporary_nam... (130 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sv5l5.grants (principal STRING NOT NULL, action_type STRING NOT N... (152 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sv5l5.grants (principal, action_type, catalog, database, table, view, udf, any_f... (118 more bytes)
07:21 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.tTKC/config.yml) doesn't exist.
07:21 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:21 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:21 INFO [databricks.labs.ucx.install] Fetching installations...
07:21 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.tTKC is corrupted. Skipping...
07:21 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:21 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:21 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:21 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:21 INFO [databricks.labs.ucx.install] Installing UCX v0.23.1+220240509072125
07:21 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sv5l5
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.external_locations (location STRING NOT NULL... (40 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:21 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sv5l5.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:22 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sv5l5.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:22 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.tTKC/README for the next steps.
07:22 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/682900195538395
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/zLEr
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1715239227924, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cyajz', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cyajz', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1715239227924, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:42 INFO [databricks.labs.ucx.install] Deleting UCX v0.23.1+220240509074222 from https://DATABRICKS_HOST
07:42 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sv5l5
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sv5l5 CASCADE
07:42 INFO [databricks.labs.ucx.install] Deleting jobs
07:42 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=482028105939691.
07:42 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=511298912877430.
07:42 INFO [databricks.labs.ucx.install] Deleting failing job_id=687365885214070.
07:42 INFO [databricks.labs.ucx.install] Deleting assessment job_id=584804769340857.
07:42 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=856322906773187.
07:42 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=924110300613503.
07:42 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=176747109887273.
07:42 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=682900195538395.
07:42 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=767714218889693.
07:42 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=83150375833725.
07:42 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=179706073051636.
07:42 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:42 INFO [databricks.labs.ucx.install] Deleting secret scope
07:42 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_tt63y', metastore_id=None, name='ucx_tt63y', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qpc8t/ucx_tt63y', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_qpc8t.ucx_tt63y
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_t4pzy', metastore_id=None, name='ucx_t4pzy', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_Ear2', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_qpc8t.ucx_t4pzy
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_t1tgx', metastore_id=None, name='ucx_t1tgx', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/zLEr', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_qpc8t.ucx_t1tgx
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_thzzk', metastore_id=None, name='ucx_thzzk', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_tt63y', view_dependencies=None)
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_qpc8t.ucx_thzzk
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_thzzk', metastore_id=None, name='ucx_thzzk', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_tt63y', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_qpc8t.ucx_thzzk is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_ttqro', metastore_id=None, name='ucx_ttqro', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_thzzk', view_dependencies=None)
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_qpc8t.ucx_ttqro
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qpc8t.ucx_ttqro', metastore_id=None, name='ucx_ttqro', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_qpc8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qpc8t.ucx_thzzk', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_qpc8t.ucx_ttqro is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qpc8t', metastore_id=None, name='migrate_qpc8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.migrate_qpc8t CASCADE
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cyajz', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cyajz.migrate_qpc8t', metastore_id=None, name='migrate_qpc8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS ucx_cyajz.migrate_qpc8t CASCADE
07:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sv5l5', metastore_id=None, name='ucx_sv5l5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:42 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sv5l5 CASCADE
[gw5] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #55}

from ucx.

github-actions commented on June 3, 2024

❌ test_migration_job_ext_hms[regular]: TimeoutError: timed out after 0:20:00: (29m11.528s)

TimeoutError: timed out after 0:20:00:
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/roIp
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.migrate_bphnb
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_bphnb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_bphnb', metastore_id=None, name='migrate_bphnb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_tepbx (id INT, value STRING)
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_tepbx: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_tepbx
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_tepbx', metastore_id=None, name='ucx_tepbx', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_bphnb/ucx_tepbx', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_te832 USING json location 'dbfs:/tmp/ucx_test_7b1h... (74 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_te832: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_te832
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_te832', metastore_id=None, name='ucx_te832', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_7b1h', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_tloxs USING CSV OPTIONS (header=true) LOCATION 'db... (24 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_tloxs: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_tloxs
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_tloxs', metastore_id=None, name='ucx_tloxs', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/roIp', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_bphnb.ucx_t7b7l AS SELECT * FROM hive_metastore.migrate_bphnb... (10 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_t7b7l: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_t7b7l
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7b7l', metastore_id=None, name='ucx_t7b7l', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_tepbx', view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_bphnb.ucx_t7u7w AS SELECT * FROM hive_metastore.migrate_bphnb... (10 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_t7u7w: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_t7u7w
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7u7w', metastore_id=None, name='ucx_t7u7w', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_t7b7l', view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE CATALOG ucx_cq0un
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1715324841477, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cq0un', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cq0un', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1715324841477, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA ucx_cq0un.migrate_bphnb
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cq0un.migrate_bphnb: https://DATABRICKS_HOST/explore/data/ucx_cq0un/migrate_bphnb
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cq0un', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cq0un.migrate_bphnb', metastore_id=None, name='migrate_bphnb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sgsb1
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgsb1: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgsb1
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgsb1', metastore_id=None, name='ucx_sgsb1', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.tables (catalog STRING NOT NULL, database STRING NOT NULL, ... (207 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.tables (catalog, database, name, object_type, table_format, location, view... (736 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.mounts (name STRING NOT NULL, source STRING NOT NULL) USING... (6 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.mounts (name, source) VALUES ('/mnt/TEST_MOUNT_NAME/a', 'abfss://TEST_MOUNT_NAME@labsCLOUD_ENVth... (29 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.groups (id_in_workspace STRING NOT NULL, name_in_workspace ... (164 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.groups (id_in_workspace, name_in_workspace, name_in_account, temporary_nam... (130 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.grants (principal STRING NOT NULL, action_type STRING NOT N... (152 more bytes)
07:08 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.grants (principal, action_type, catalog, database, table, view, udf, any_f... (118 more bytes)
[gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/roIp
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.migrate_bphnb
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_bphnb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_bphnb', metastore_id=None, name='migrate_bphnb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_tepbx (id INT, value STRING)
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_tepbx: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_tepbx
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_tepbx', metastore_id=None, name='ucx_tepbx', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_bphnb/ucx_tepbx', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_te832 USING json location 'dbfs:/tmp/ucx_test_7b1h... (74 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_te832: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_te832
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_te832', metastore_id=None, name='ucx_te832', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_7b1h', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_tloxs USING CSV OPTIONS (header=true) LOCATION 'db... (24 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_tloxs: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_tloxs
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_tloxs', metastore_id=None, name='ucx_tloxs', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/roIp', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_bphnb.ucx_t7b7l AS SELECT * FROM hive_metastore.migrate_bphnb... (10 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_t7b7l: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_t7b7l
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7b7l', metastore_id=None, name='ucx_t7b7l', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_tepbx', view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_bphnb.ucx_t7u7w AS SELECT * FROM hive_metastore.migrate_bphnb... (10 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_t7u7w: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_t7u7w
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7u7w', metastore_id=None, name='ucx_t7u7w', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_t7b7l', view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE CATALOG ucx_cq0un
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1715324841477, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cq0un', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cq0un', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1715324841477, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA ucx_cq0un.migrate_bphnb
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cq0un.migrate_bphnb: https://DATABRICKS_HOST/explore/data/ucx_cq0un/migrate_bphnb
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cq0un', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cq0un.migrate_bphnb', metastore_id=None, name='migrate_bphnb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sgsb1
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgsb1: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgsb1
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgsb1', metastore_id=None, name='ucx_sgsb1', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.tables (catalog STRING NOT NULL, database STRING NOT NULL, ... (207 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.tables (catalog, database, name, object_type, table_format, location, view... (736 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.mounts (name STRING NOT NULL, source STRING NOT NULL) USING... (6 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.mounts (name, source) VALUES ('/mnt/TEST_MOUNT_NAME/a', 'abfss://TEST_MOUNT_NAME@labsCLOUD_ENVth... (29 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.groups (id_in_workspace STRING NOT NULL, name_in_workspace ... (164 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.groups (id_in_workspace, name_in_workspace, name_in_account, temporary_nam... (130 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.grants (principal STRING NOT NULL, action_type STRING NOT N... (152 more bytes)
07:08 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.grants (principal, action_type, catalog, database, table, view, udf, any_f... (118 more bytes)
07:08 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.0vNB/config.yml) doesn't exist.
07:08 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:08 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:08 INFO [databricks.labs.ucx.install] Fetching installations...
07:08 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.0vNB is corrupted. Skipping...
07:08 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:08 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:08 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:13 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:13 INFO [databricks.labs.ucx.install] Installing UCX v0.23.1+320240510071335
07:13 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sgsb1
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.external_locations (location STRING NOT NULL... (40 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:14 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.0vNB/README for the next steps.
07:14 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/266648573014447
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/roIp
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.migrate_bphnb
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_bphnb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_bphnb', metastore_id=None, name='migrate_bphnb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_tepbx (id INT, value STRING)
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_tepbx: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_tepbx
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_tepbx', metastore_id=None, name='ucx_tepbx', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_bphnb/ucx_tepbx', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_te832 USING json location 'dbfs:/tmp/ucx_test_7b1h... (74 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_te832: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_te832
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_te832', metastore_id=None, name='ucx_te832', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_7b1h', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE hive_metastore.migrate_bphnb.ucx_tloxs USING CSV OPTIONS (header=true) LOCATION 'db... (24 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_tloxs: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_tloxs
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_tloxs', metastore_id=None, name='ucx_tloxs', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/roIp', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_bphnb.ucx_t7b7l AS SELECT * FROM hive_metastore.migrate_bphnb... (10 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_t7b7l: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_t7b7l
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7b7l', metastore_id=None, name='ucx_t7b7l', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_tepbx', view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE VIEW hive_metastore.migrate_bphnb.ucx_t7u7w AS SELECT * FROM hive_metastore.migrate_bphnb... (10 more bytes)
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_bphnb.ucx_t7u7w: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_bphnb/ucx_t7u7w
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7u7w', metastore_id=None, name='ucx_t7u7w', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_t7b7l', view_dependencies=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE CATALOG ucx_cq0un
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1715324841477, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cq0un', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cq0un', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1715324841477, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA ucx_cq0un.migrate_bphnb
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cq0un.migrate_bphnb: https://DATABRICKS_HOST/explore/data/ucx_cq0un/migrate_bphnb
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cq0un', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cq0un.migrate_bphnb', metastore_id=None, name='migrate_bphnb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sgsb1
07:07 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgsb1: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgsb1
07:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgsb1', metastore_id=None, name='ucx_sgsb1', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.tables (catalog STRING NOT NULL, database STRING NOT NULL, ... (207 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.tables (catalog, database, name, object_type, table_format, location, view... (736 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.mounts (name STRING NOT NULL, source STRING NOT NULL) USING... (6 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.mounts (name, source) VALUES ('/mnt/TEST_MOUNT_NAME/a', 'abfss://TEST_MOUNT_NAME@labsCLOUD_ENVth... (29 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.groups (id_in_workspace STRING NOT NULL, name_in_workspace ... (164 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.groups (id_in_workspace, name_in_workspace, name_in_account, temporary_nam... (130 more bytes)
07:07 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS ucx_sgsb1.grants (principal STRING NOT NULL, action_type STRING NOT N... (152 more bytes)
07:08 DEBUG [tests.integration.test_ext_hms] [api][execute] INSERT INTO ucx_sgsb1.grants (principal, action_type, catalog, database, table, view, udf, any_f... (118 more bytes)
07:08 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.0vNB/config.yml) doesn't exist.
07:08 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:08 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:08 INFO [databricks.labs.ucx.install] Fetching installations...
07:08 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.0vNB is corrupted. Skipping...
07:08 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:08 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:08 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:13 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:13 INFO [databricks.labs.ucx.install] Installing UCX v0.23.1+320240510071335
07:13 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sgsb1
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.external_locations (location STRING NOT NULL... (40 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:13 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sgsb1.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:14 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sgsb1.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:14 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:14 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.0vNB/README for the next steps.
07:14 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/266648573014447
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/roIp
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1715324841477, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cq0un', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cq0un', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1715324841477, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:34 INFO [databricks.labs.ucx.install] Deleting UCX v0.23.1+320240510073447 from https://DATABRICKS_HOST
07:34 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sgsb1
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sgsb1 CASCADE
07:34 INFO [databricks.labs.ucx.install] Deleting jobs
07:34 INFO [databricks.labs.ucx.install] Deleting assessment job_id=135542168341941.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=4500871854460.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=207644356295356.
07:34 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=809362262868490.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=734016583929312.
07:34 INFO [databricks.labs.ucx.install] Deleting failing job_id=1002636313399117.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=686691851313058.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=656626457907666.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=266648573014447.
07:34 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=767341219775284.
07:34 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=424482265834425.
07:34 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:34 INFO [databricks.labs.ucx.install] Deleting secret scope
07:34 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_tepbx', metastore_id=None, name='ucx_tepbx', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_bphnb/ucx_tepbx', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_bphnb.ucx_tepbx
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_te832', metastore_id=None, name='ucx_te832', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_7b1h', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_bphnb.ucx_te832
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_tloxs', metastore_id=None, name='ucx_tloxs', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/roIp', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_bphnb.ucx_tloxs
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7b7l', metastore_id=None, name='ucx_t7b7l', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_tepbx', view_dependencies=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_bphnb.ucx_t7b7l
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7b7l', metastore_id=None, name='ucx_t7b7l', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_tepbx', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_bphnb.ucx_t7b7l is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7u7w', metastore_id=None, name='ucx_t7u7w', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_t7b7l', view_dependencies=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP TABLE IF EXISTS hive_metastore.migrate_bphnb.ucx_t7u7w
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_bphnb.ucx_t7u7w', metastore_id=None, name='ucx_t7u7w', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='migrate_bphnb', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_bphnb.ucx_t7b7l', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_bphnb.ucx_t7u7w is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_bphnb', metastore_id=None, name='migrate_bphnb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.migrate_bphnb CASCADE
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cq0un', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cq0un.migrate_bphnb', metastore_id=None, name='migrate_bphnb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:35 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS ucx_cq0un.migrate_bphnb CASCADE
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgsb1', metastore_id=None, name='ucx_sgsb1', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:35 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sgsb1 CASCADE
[gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #56}

from ucx.

Test failure: `test_migration_job_ext_hms[regular]` about ucx HOT 2 CLOSED

Comments (2)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent