Accessing Azure Data Lake Gen 2 Storage about docker-trino-cluster

[eddingtonm]

Hi again,

What I've tried to configure is a hive connector within a hive.properties file under the example/etc/catalog path with a file based metastore for test purposes as well as the Azure configs for ADLS gen 2 storage:

connector.name=hive-hadoop2 hive.metastore=file hive.metastore.catalog.dir=file:///tmp/hive_catalog hive.azure.abfs-storage-account=<name of storage account> hive.azure.abfs-access-key=<access key>

I am able to create a hive based schema from the presto-cli tool within the coordinator container but when I try to create a table based on an external location as follows:

CREATE TABLE OrderStats ( ID int, orderdate date, quantity bigint .... ) WITH ( external_location = 'abfs://container_name@storage_account.dfs.core.windows.net/some_folder/file_folder', format = 'PARQUET' );

It gives out an error: Query failed: Failed checking path: abfs://container_name@storage_account.dfs.core.windows.net/some_folder/file_folder

I tried using the 341-snapshot release as well as the 340 release both without success.

Could it have to do with Docker itself?

Eddie

[eddingtonm]

Hi,

I managed to get it to work by adding a parallel Hive Docker setup.

The version of Hadoop should be >= 3.2.0 for Azure Gen2 storage. I used Hive version 3.1.2 and one would need a bunch of jar files such as azure-storage and azure-datalake.

This could be added to the docker-compose.yml setup to make it one config setup.

Hope this helps.

Eddie