Comments (2)
github-actions
commented on June 12, 2024
❌ test_running_real_assessment_job_ext_hms: AssertionError: assert False (21m59.917s)
AssertionError: assert False
+ where False = <bound method DeployedWorkflows.validate_step of <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7f25e83d2620>>('assessment')
+ where <bound method DeployedWorkflows.validate_step of <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7f25e83d2620>> = <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7f25e83d2620>.validate_step
+ where <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7f25e83d2620> = <tests.integration.conftest.TestInstallationContext object at 0x7f25e839bf70>.deployed_workflows
[gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='6274411575154155', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GZnzG: https://DATABRICKS_HOST#setting/accounts/groups/543294494725189
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GZnzG', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='543294494725189', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/6274411575154155', type=None, value='6274411575154155')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GZnzG: https://accounts.CLOUD_ENVdatabricks.net/users/groups/429088899187918/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GZnzG', entitlements=[], external_id=None, groups=[], id='429088899187918', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/6274411575154155', type=None, value='6274411575154155')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0005B5B7138F288E
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0005B5B7138F288E')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0005B5B7138F288E [group_name admins CAN_USE] -> [group_name ucx_GZnzG CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sfemk
07:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sfemk: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sfemk
07:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sfemk', metastore_id=None, name='ucx_sfemk', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:16 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.IlXw/config.yml) doesn't exist.
07:16 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:16 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:16 INFO [databricks.labs.ucx.install] Fetching installations...
07:16 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 INFO [databricks.labs.ucx.install] Installing UCX v0.23.2+120240512071645
07:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sfemk
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.external_locations (location STRING NOT NULL... (40 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:18 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.IlXw/README for the next steps.
07:18 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/452536021066987
07:26 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/452536021066987
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='6274411575154155', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GZnzG: https://DATABRICKS_HOST#setting/accounts/groups/543294494725189
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GZnzG', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='543294494725189', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/6274411575154155', type=None, value='6274411575154155')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GZnzG: https://accounts.CLOUD_ENVdatabricks.net/users/groups/429088899187918/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GZnzG', entitlements=[], external_id=None, groups=[], id='429088899187918', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/6274411575154155', type=None, value='6274411575154155')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0005B5B7138F288E
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0005B5B7138F288E')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0005B5B7138F288E [group_name admins CAN_USE] -> [group_name ucx_GZnzG CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sfemk
07:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sfemk: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sfemk
07:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sfemk', metastore_id=None, name='ucx_sfemk', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:16 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.IlXw/config.yml) doesn't exist.
07:16 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:16 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:16 INFO [databricks.labs.ucx.install] Fetching installations...
07:16 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 INFO [databricks.labs.ucx.install] Installing UCX v0.23.2+120240512071645
07:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sfemk
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.external_locations (location STRING NOT NULL... (40 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sfemk.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sfemk.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:18 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.IlXw/README for the next steps.
07:18 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/452536021066987
07:26 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/452536021066987
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 0005B5B7138F288E [group_name admins CAN_USE] -> [group_name ucx_GZnzG CAN_USE]
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='0005B5B7138F288E')
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='0005B5B7138F288E') teardown: Can't find a cluster policy with id: 0005B5B7138F288E.
07:26 INFO [databricks.labs.ucx.install] Deleting UCX v0.23.2+120240512072642 from https://DATABRICKS_HOST
07:26 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sfemk
07:26 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sfemk CASCADE
07:26 INFO [databricks.labs.ucx.install] Deleting jobs
07:26 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=809348872434107.
07:26 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=963669200263419.
07:26 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=213909353213459.
07:26 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=944924465157231.
07:26 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=366939562209388.
07:26 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=1046625304222798.
07:26 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=368263514479906.
07:26 INFO [databricks.labs.ucx.install] Deleting assessment job_id=452536021066987.
07:26 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=732946268665515.
07:26 INFO [databricks.labs.ucx.install] Deleting failing job_id=925212035962551.
07:26 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=979327290444633.
07:26 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:26 INFO [databricks.labs.ucx.install] Deleting secret scope
07:26 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='6274411575154155', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='6274411575154155', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]') teardown: None User with id 6274411575154155 not found.
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_GZnzG', entitlements=[], external_id=None, groups=[], id='429088899187918', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/6274411575154155', type=None, value='6274411575154155')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_GZnzG', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='543294494725189', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/6274411575154155', type=None, value='6274411575154155')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_GZnzG', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='543294494725189', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/6274411575154155', type=None, value='6274411575154155')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 543294494725189 not found.
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sfemk', metastore_id=None, name='ucx_sfemk', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:26 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sfemk CASCADE
[gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
Running from nightly #58
from ucx.
github-actions
commented on June 12, 2024
❌ test_running_real_assessment_job_ext_hms: AssertionError: assert False (6m53.006s)
AssertionError: assert False
+ where False = <bound method DeployedWorkflows.validate_step of <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7f31dc7806d0>>('assessment')
+ where <bound method DeployedWorkflows.validate_step of <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7f31dc7806d0>> = <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7f31dc7806d0>.validate_step
+ where <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7f31dc7806d0> = <tests.integration.conftest.TestInstallationContext object at 0x7f31dc78fa30>.deployed_workflows
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='2653671575168927', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_G7cnU: https://DATABRICKS_HOST#setting/accounts/groups/964038840499789
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_G7cnU', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='964038840499789', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/2653671575168927', type=None, value='2653671575168927')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_G7cnU: https://accounts.CLOUD_ENVdatabricks.net/users/groups/888986743016372/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_G7cnU', entitlements=[], external_id=None, groups=[], id='888986743016372', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/2653671575168927', type=None, value='2653671575168927')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000CD3A8AFD7DBC1
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000CD3A8AFD7DBC1')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000CD3A8AFD7DBC1 [group_name admins CAN_USE] -> [group_name ucx_G7cnU CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sebsm
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sebsm: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sebsm
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sebsm', metastore_id=None, name='ucx_sebsm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:05 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Xh8h/config.yml) doesn't exist.
07:05 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:05 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:05 INFO [databricks.labs.ucx.install] Fetching installations...
07:05 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:05 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:05 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:05 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:05 INFO [databricks.labs.ucx.install] Installing UCX v0.23.2+120240513070546
07:05 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sebsm
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.external_locations (location STRING NOT NULL... (40 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:06 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Xh8h/README for the next steps.
07:06 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/692738058956860
07:12 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/692738058956860
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='2653671575168927', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_G7cnU: https://DATABRICKS_HOST#setting/accounts/groups/964038840499789
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_G7cnU', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='964038840499789', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/2653671575168927', type=None, value='2653671575168927')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_G7cnU: https://accounts.CLOUD_ENVdatabricks.net/users/groups/888986743016372/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_G7cnU', entitlements=[], external_id=None, groups=[], id='888986743016372', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/2653671575168927', type=None, value='2653671575168927')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000CD3A8AFD7DBC1
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000CD3A8AFD7DBC1')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000CD3A8AFD7DBC1 [group_name admins CAN_USE] -> [group_name ucx_G7cnU CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sebsm
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sebsm: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sebsm
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sebsm', metastore_id=None, name='ucx_sebsm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:05 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Xh8h/config.yml) doesn't exist.
07:05 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:05 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:05 INFO [databricks.labs.ucx.install] Fetching installations...
07:05 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:05 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:05 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:05 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:05 INFO [databricks.labs.ucx.install] Installing UCX v0.23.2+120240513070546
07:05 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sebsm
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.external_locations (location STRING NOT NULL... (40 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sebsm.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sebsm.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:06 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Xh8h/README for the next steps.
07:06 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/692738058956860
07:12 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/692738058956860
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 000CD3A8AFD7DBC1 [group_name admins CAN_USE] -> [group_name ucx_G7cnU CAN_USE]
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='000CD3A8AFD7DBC1')
07:12 INFO [databricks.labs.ucx.install] Deleting UCX v0.23.2+120240513071212 from https://DATABRICKS_HOST
07:12 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sebsm
07:12 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sebsm CASCADE
07:12 INFO [databricks.labs.ucx.install] Deleting jobs
07:12 INFO [databricks.labs.ucx.install] Deleting assessment job_id=692738058956860.
07:12 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=461626176519673.
07:12 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=737910437704533.
07:12 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=517101868174056.
07:12 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=358239012233562.
07:12 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=533222893633016.
07:12 INFO [databricks.labs.ucx.install] Deleting failing job_id=632800322875292.
07:12 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=765096651477105.
07:12 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=538187480253500.
07:12 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=144457687471334.
07:12 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=469656835666001.
07:12 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:12 INFO [databricks.labs.ucx.install] Deleting secret scope
07:12 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='2653671575168927', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_G7cnU', entitlements=[], external_id=None, groups=[], id='888986743016372', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/2653671575168927', type=None, value='2653671575168927')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_G7cnU', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='964038840499789', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/2653671575168927', type=None, value='2653671575168927')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sebsm', metastore_id=None, name='ucx_sebsm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:12 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sebsm CASCADE
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
Running from nightly #59
from ucx.
Related Issues (20)
- [BUG]: Acceptance tests cannot run for PR raised from forks HOT 1
- [BUG]: AWS Migrate Credentials Doesn't work properly
- [FEATURE]: Use fixed test coverage thresholds
- [FEATURE]: Refactor `PipCell` into a `PipCommand`
- [FEATURE]: Dashboard for post-migration reconciliation results
- Test failure: `test_running_real_workflow_linter_job`
- Test failure: `test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]` HOT 1
- [FEATURE] Clickable links in code migration dashboard creation
- [BUG]: Upgrade from 0.16 to latest has error 'No module named 'databricks.labs.lsql' HOT 2
- [BUG]: lint-local-code raises numerous pointless advices
- Test failure: `test_linter_from_context`
- [BUG]: `PipResolver_locate_dist_info` matches multiple `.dist-info`'s
- [FEATURE]: Table migration process only logs errors to files, but not to migration status tables for further analysis HOT 1
- [BUG]: ucx install Fails on Azure with AWS-linked validation error HOT 1
- Test failure: `test_table_migration_job_refreshes_migration_status[regular-migrate-tables]`
- Test failure: `test_running_real_assessment_job_ext_hms`
- [BUG]: sync-workspace-info does not find workspace endpoint HOT 2
- [BUG]: parse_logs yields AssertionError HOT 1
- [BUG]: create-table-mapping fails for unsuccesful sync-workspace-info attemps HOT 1
- Test failure: `test_linter_from_context`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ucx.