❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_tnIh' (25m7.398s)</sum

Running this integration test in isolation works on an up-to-date <code class="notrans

Test failure: `test_running_real_assessment_job_ext_hms`,about databrickslabs/ucx

Comments (7)

github-actions commented on June 12, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_ORS8' (25m36.24s)

KeyError: 'ucx_ORS8'
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='49835715119411', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_ORS8: https://DATABRICKS_HOST#setting/accounts/groups/282498917932873
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_ORS8', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='282498917932873', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_ORS8: https://accounts.CLOUD_ENVdatabricks.net/users/groups/903899026305652/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_ORS8', entitlements=[], external_id=None, groups=[], id='903899026305652', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/00183E02C8D02B37
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='00183E02C8D02B37')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 00183E02C8D02B37 [group_name admins CAN_USE] -> [group_name ucx_ORS8 CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sypx9
07:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sypx9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sypx9
07:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sypx9', metastore_id=None, name='ucx_sypx9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:16 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.neCs/config.yml) doesn't exist.
07:16 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:16 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:16 INFO [databricks.labs.ucx.install] Fetching installations...
07:16 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240504071657
07:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sypx9
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.external_locations (location STRING NOT NULL... (40 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:18 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.neCs/README for the next steps.
07:18 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1056246959260187
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1056246959260187
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='49835715119411', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_ORS8: https://DATABRICKS_HOST#setting/accounts/groups/282498917932873
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_ORS8', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='282498917932873', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_ORS8: https://accounts.CLOUD_ENVdatabricks.net/users/groups/903899026305652/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_ORS8', entitlements=[], external_id=None, groups=[], id='903899026305652', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/00183E02C8D02B37
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='00183E02C8D02B37')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 00183E02C8D02B37 [group_name admins CAN_USE] -> [group_name ucx_ORS8 CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sypx9
07:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sypx9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sypx9
07:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sypx9', metastore_id=None, name='ucx_sypx9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:16 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.neCs/config.yml) doesn't exist.
07:16 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:16 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:16 INFO [databricks.labs.ucx.install] Fetching installations...
07:16 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240504071657
07:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sypx9
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.external_locations (location STRING NOT NULL... (40 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:18 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.neCs/README for the next steps.
07:18 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1056246959260187
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1056246959260187
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 00183E02C8D02B37 [group_name admins CAN_USE] -> [group_name ucx_ORS8 CAN_USE]
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='00183E02C8D02B37')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='00183E02C8D02B37') teardown: Can't find a cluster policy with id: 00183E02C8D02B37.
07:30 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+3220240504073003 from https://DATABRICKS_HOST
07:30 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sypx9
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sypx9 CASCADE
07:30 INFO [databricks.labs.ucx.install] Deleting jobs
07:30 INFO [databricks.labs.ucx.install] Deleting failing job_id=587841249730879.
07:30 INFO [databricks.labs.ucx.install] Deleting assessment job_id=1056246959260187.
07:30 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=409633535547014.
07:30 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=199580646083585.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=912505395576906.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=174812062070925.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1387661723379.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=80407497854529.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=382570426554417.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=75205767086363.
07:30 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:30 INFO [databricks.labs.ucx.install] Deleting secret scope
07:30 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='49835715119411', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='49835715119411', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]') teardown: None User with id 49835715119411 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_ORS8', entitlements=[], external_id=None, groups=[], id='903899026305652', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_ORS8', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='282498917932873', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_ORS8', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='282498917932873', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 282498917932873 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sypx9', metastore_id=None, name='ucx_sypx9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sypx9 CASCADE
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #50}

from ucx.

github-actions commented on June 12, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_wWSV' (24m44.187s)

KeyError: 'ucx_wWSV'
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='3753449921302703', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_wWSV: https://DATABRICKS_HOST#setting/accounts/groups/599730207407029
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_wWSV', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='599730207407029', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_wWSV: https://accounts.CLOUD_ENVdatabricks.net/users/groups/806626359434077/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_wWSV', entitlements=[], external_id=None, groups=[], id='806626359434077', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/001342C3CE3F5A01
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='001342C3CE3F5A01')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 001342C3CE3F5A01 [group_name admins CAN_USE] -> [group_name ucx_wWSV CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sjzkr
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sjzkr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sjzkr
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sjzkr', metastore_id=None, name='ucx_sjzkr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Dg3s/config.yml) doesn't exist.
07:15 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:15 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:15 INFO [databricks.labs.ucx.install] Fetching installations...
07:15 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:15 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240505071556
07:15 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sjzkr
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.external_locations (location STRING NOT NULL... (40 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:17 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Dg3s/README for the next steps.
07:17 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/255392742700621
07:28 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/255392742700621
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='3753449921302703', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_wWSV: https://DATABRICKS_HOST#setting/accounts/groups/599730207407029
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_wWSV', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='599730207407029', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_wWSV: https://accounts.CLOUD_ENVdatabricks.net/users/groups/806626359434077/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_wWSV', entitlements=[], external_id=None, groups=[], id='806626359434077', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/001342C3CE3F5A01
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='001342C3CE3F5A01')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 001342C3CE3F5A01 [group_name admins CAN_USE] -> [group_name ucx_wWSV CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sjzkr
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sjzkr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sjzkr
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sjzkr', metastore_id=None, name='ucx_sjzkr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Dg3s/config.yml) doesn't exist.
07:15 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:15 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:15 INFO [databricks.labs.ucx.install] Fetching installations...
07:15 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:15 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240505071556
07:15 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sjzkr
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.external_locations (location STRING NOT NULL... (40 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:17 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Dg3s/README for the next steps.
07:17 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/255392742700621
07:28 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/255392742700621
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 001342C3CE3F5A01 [group_name admins CAN_USE] -> [group_name ucx_wWSV CAN_USE]
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='001342C3CE3F5A01')
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='001342C3CE3F5A01') teardown: Can't find a cluster policy with id: 001342C3CE3F5A01.
07:28 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+3220240505072853 from https://DATABRICKS_HOST
07:28 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sjzkr
07:28 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sjzkr CASCADE
07:28 INFO [databricks.labs.ucx.install] Deleting jobs
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=803662989067337.
07:28 INFO [databricks.labs.ucx.install] Deleting assessment job_id=255392742700621.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=574141874107506.
07:28 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=828472765882381.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=596970312271364.
07:28 INFO [databricks.labs.ucx.install] Deleting failing job_id=125935612576049.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=235375964036838.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=369025870963211.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=127488673308845.
07:28 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=175964184317291.
07:28 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:28 INFO [databricks.labs.ucx.install] Deleting secret scope
07:28 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='3753449921302703', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='3753449921302703', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]') teardown: None User with id 3753449921302703 not found.
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_wWSV', entitlements=[], external_id=None, groups=[], id='806626359434077', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_wWSV', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='599730207407029', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_wWSV', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='599730207407029', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 599730207407029 not found.
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sjzkr', metastore_id=None, name='ucx_sjzkr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:29 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sjzkr CASCADE
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #51}

from ucx.

github-actions commented on June 12, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_D5jw' (25m26.539s)

KeyError: 'ucx_D5jw'
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5593186882414224', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_D5jw: https://DATABRICKS_HOST#setting/accounts/groups/430512368140466
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_D5jw', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='430512368140466', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_D5jw: https://accounts.CLOUD_ENVdatabricks.net/users/groups/363376592364516/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_D5jw', entitlements=[], external_id=None, groups=[], id='363376592364516', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000796608048429E
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000796608048429E')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000796608048429E [group_name admins CAN_USE] -> [group_name ucx_D5jw CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_smrhe
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_smrhe: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_smrhe
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smrhe', metastore_id=None, name='ucx_smrhe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ckVZ/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240506071809
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_smrhe
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:20 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ckVZ/README for the next steps.
07:20 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1020303679279872
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1020303679279872
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5593186882414224', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_D5jw: https://DATABRICKS_HOST#setting/accounts/groups/430512368140466
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_D5jw', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='430512368140466', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_D5jw: https://accounts.CLOUD_ENVdatabricks.net/users/groups/363376592364516/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_D5jw', entitlements=[], external_id=None, groups=[], id='363376592364516', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000796608048429E
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000796608048429E')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000796608048429E [group_name admins CAN_USE] -> [group_name ucx_D5jw CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_smrhe
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_smrhe: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_smrhe
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smrhe', metastore_id=None, name='ucx_smrhe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ckVZ/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240506071809
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_smrhe
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:20 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ckVZ/README for the next steps.
07:20 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1020303679279872
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1020303679279872
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 000796608048429E [group_name admins CAN_USE] -> [group_name ucx_D5jw CAN_USE]
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='000796608048429E')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='000796608048429E') teardown: Can't find a cluster policy with id: 000796608048429E.
07:30 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+3220240506073023 from https://DATABRICKS_HOST
07:30 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_smrhe
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_smrhe CASCADE
07:30 INFO [databricks.labs.ucx.install] Deleting jobs
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=647769373469020.
07:30 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=985369021765508.
07:30 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=412118750744371.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=822388136460936.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=1088464868943184.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=76918293174651.
07:30 INFO [databricks.labs.ucx.install] Deleting failing job_id=839658168553027.
07:30 INFO [databricks.labs.ucx.install] Deleting assessment job_id=1020303679279872.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=995324220637486.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=476358037385030.
07:30 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:30 INFO [databricks.labs.ucx.install] Deleting secret scope
07:30 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5593186882414224', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5593186882414224', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]') teardown: None User with id 5593186882414224 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_D5jw', entitlements=[], external_id=None, groups=[], id='363376592364516', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_D5jw', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='430512368140466', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_D5jw', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='430512368140466', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 430512368140466 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smrhe', metastore_id=None, name='ucx_smrhe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_smrhe CASCADE
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #52}

from ucx.

JCZuurmond commented on June 12, 2024

Running this integration test in isolation works on an up-to-date main, let's wait another to see if the issue remains.

from ucx.

github-actions commented on June 12, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_GIu4b' (28m11.568s)

KeyError: 'ucx_GIu4b'
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='4059398050832766', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GIu4b: https://DATABRICKS_HOST#setting/accounts/groups/1011098775732898
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GIu4b', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1011098775732898', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GIu4b: https://accounts.CLOUD_ENVdatabricks.net/users/groups/111040381670897/members
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GIu4b', entitlements=[], external_id=None, groups=[], id='111040381670897', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0006C10A86B07D44
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0006C10A86B07D44')
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0006C10A86B07D44 [group_name admins CAN_USE] -> [group_name ucx_GIu4b CAN_USE]
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_s7f5d
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s7f5d: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s7f5d
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s7f5d', metastore_id=None, name='ucx_s7f5d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XZNr/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+5920240508071846
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_s7f5d
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:21 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XZNr/README for the next steps.
07:21 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/763983188280201
07:34 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/763983188280201
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='4059398050832766', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GIu4b: https://DATABRICKS_HOST#setting/accounts/groups/1011098775732898
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GIu4b', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1011098775732898', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GIu4b: https://accounts.CLOUD_ENVdatabricks.net/users/groups/111040381670897/members
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GIu4b', entitlements=[], external_id=None, groups=[], id='111040381670897', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0006C10A86B07D44
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0006C10A86B07D44')
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0006C10A86B07D44 [group_name admins CAN_USE] -> [group_name ucx_GIu4b CAN_USE]
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_s7f5d
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s7f5d: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s7f5d
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s7f5d', metastore_id=None, name='ucx_s7f5d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XZNr/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+5920240508071846
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_s7f5d
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:21 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XZNr/README for the next steps.
07:21 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/763983188280201
07:34 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/763983188280201
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 0006C10A86B07D44 [group_name admins CAN_USE] -> [group_name ucx_GIu4b CAN_USE]
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='0006C10A86B07D44')
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='0006C10A86B07D44') teardown: Can't find a cluster policy with id: 0006C10A86B07D44.
07:34 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+5920240508073412 from https://DATABRICKS_HOST
07:34 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_s7f5d
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_s7f5d CASCADE
07:34 INFO [databricks.labs.ucx.install] Deleting jobs
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=508300475131428.
07:34 INFO [databricks.labs.ucx.install] Deleting failing job_id=166634836047744.
07:34 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=226671084118256.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=336203416402049.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=146741978055920.
07:34 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=680320723792549.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1061588380583787.
07:34 INFO [databricks.labs.ucx.install] Deleting assessment job_id=763983188280201.
07:34 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=421491137745054.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=930913717937964.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=149368825494921.
07:34 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:34 INFO [databricks.labs.ucx.install] Deleting secret scope
07:34 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='4059398050832766', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='4059398050832766', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]') teardown: None User with id 4059398050832766 not found.
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_GIu4b', entitlements=[], external_id=None, groups=[], id='111040381670897', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_GIu4b', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1011098775732898', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_GIu4b', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1011098775732898', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 1011098775732898 not found.
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s7f5d', metastore_id=None, name='ucx_s7f5d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_s7f5d CASCADE
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #54}

from ucx.

github-actions commented on June 12, 2024

❌ test_running_real_assessment_job_ext_hms: AssertionError: Group ucx_Gfrhs not found in cluster policy (29m25.923s)

AssertionError: Group ucx_Gfrhs not found in cluster policy
assert 'ucx_Gfrhs' in {'0a330eb5-dd51-4d97-b6e4-c474356b1d5d': <PermissionLevel.CAN_USE: 'CAN_USE'>}
 +  where 'ucx_Gfrhs' = Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='all...[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]).display_name
[gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7535334714784639', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_Gfrhs: https://DATABRICKS_HOST#setting/accounts/groups/898539675448914
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='898539675448914', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_Gfrhs: https://accounts.CLOUD_ENVdatabricks.net/users/groups/70908068725976/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_Gfrhs', entitlements=[], external_id=None, groups=[], id='70908068725976', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0006678BE3CF7725
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0006678BE3CF7725')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0006678BE3CF7725 [group_name admins CAN_USE] -> [group_name ucx_Gfrhs CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sn6ow
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sn6ow: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sn6ow
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sn6ow', metastore_id=None, name='ucx_sn6ow', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.LGMU/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.23.1+220240509071855
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sn6ow
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:21 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.LGMU/README for the next steps.
07:21 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/805561307152876
07:34 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/805561307152876
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7535334714784639', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_Gfrhs: https://DATABRICKS_HOST#setting/accounts/groups/898539675448914
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='898539675448914', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_Gfrhs: https://accounts.CLOUD_ENVdatabricks.net/users/groups/70908068725976/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_Gfrhs', entitlements=[], external_id=None, groups=[], id='70908068725976', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0006678BE3CF7725
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0006678BE3CF7725')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0006678BE3CF7725 [group_name admins CAN_USE] -> [group_name ucx_Gfrhs CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sn6ow
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sn6ow: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sn6ow
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sn6ow', metastore_id=None, name='ucx_sn6ow', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.LGMU/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.23.1+220240509071855
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sn6ow
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:21 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.LGMU/README for the next steps.
07:21 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/805561307152876
07:34 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/805561307152876
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 0006678BE3CF7725 [group_name admins CAN_USE] -> [group_name ucx_Gfrhs CAN_USE]
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='0006678BE3CF7725')
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='0006678BE3CF7725') teardown: Can't find a cluster policy with id: 0006678BE3CF7725.
07:34 INFO [databricks.labs.ucx.install] Deleting UCX v0.23.1+220240509073432 from https://DATABRICKS_HOST
07:34 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sn6ow
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sn6ow CASCADE
07:34 INFO [databricks.labs.ucx.install] Deleting jobs
07:34 INFO [databricks.labs.ucx.install] Deleting assessment job_id=805561307152876.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=888058956393416.
07:34 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=1120896398887648.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=946932723019634.
07:34 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=1122088821949531.
07:34 INFO [databricks.labs.ucx.install] Deleting failing job_id=734032708287244.
07:34 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=313296447930976.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=147829886125175.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=520850033632636.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=209220968508595.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=161822184494260.
07:34 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:34 INFO [databricks.labs.ucx.install] Deleting secret scope
07:34 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7535334714784639', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7535334714784639', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]') teardown: None User with id 7535334714784639 not found.
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_Gfrhs', entitlements=[], external_id=None, groups=[], id='70908068725976', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='898539675448914', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='898539675448914', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 898539675448914 not found.
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sn6ow', metastore_id=None, name='ucx_sn6ow', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sn6ow CASCADE
[gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #55}

from ucx.

JCZuurmond commented on June 12, 2024

Not present anymore in latest test run

from ucx.

Test failure: `test_running_real_assessment_job_ext_hms` about ucx HOT 7 CLOSED

Comments (7)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent