cube-js / airflow-provider-cube

This is a provider package for Airflow Cube Provider.

Airflow Cube Provider

Cube is the semantic layer for building data applications. It helps data engineers and application developers access data from modern data stores, organize it into consistent definitions, and deliver it to every application.

Connection requirements

CubeHook uses Generic connection. So, in order to create a connection to a Cube you will need to create an Airflow Generic connection and specify the following required properties:

• host - your Cube's host (must contain schema, address and port);
• password - your Cube's CUBEJS_API_SECRET;
• extra.security_context - your user securityContext (in a Cube's terms).

Rest of the connection fields will be ignored.

Note: the CubeBuildOperator uses Cube's /cubejs-api/v1/pre-aggregations/jobs endpoint that is forbiden by the default. Make sure that specified securityContext and Cube's contextToPermissions function are configured in a way that allow you to run this query. See Cube's documentation for a more context.

Package description

This package provides the common CubeHook class, the abstract CubeBaseOperator, the CubeQueryOperator to run analytical queries over a Cube and the CubeBuildOperator to run pre-aggregations build process.

Dependencies

Python 3.10

Package	Version
typing	2.6.0
jwt	2.6.0
json	2.0.9
requests	2.28.2
airflow	2.5.1

airflow-provider-cube's Issues

CubeBuild operator can hang due to lack of handling of other failure types

I believe the Jobs api can return other failures not captured by this code block resulting in it running until Airflow timeout

https://github.com/cube-js/airflow-provider-cube/blob/main/cube_provider/operators/cube.py#L198-L204

For example - other statuses seem to be "orphaned" and potentially others.

JWT logic is hard coded to use HS256

See code here:
https://github.com/cube-js/airflow-provider-cube/blob/main/cube_provider/hooks/cube.py#L92

I think it is possible to override the headers by passing in an AUTHORIZATION token - and this might make more sense - but figured I'd flag that this should probably be an argument or derive itself from the Cube configuration?

Allow passing custom endpoint to CubeBuildOperator

The CubeBuildOperator constructor uses a hardcoded endpoint. The endpoint for my use would replace the prefix to /cubejs-api/v1/pre-aggregations/jobs but I'm not able to override by setting arguments when creating an operator.

airflow-provider-cube/cube_provider/operators/cube.py

Line 153 in 99a8d88

endpoint="/cubejs-api/v1/pre-aggregations/jobs",

Continue wait errors for build operator return an internal processing error

I believe continue wait errors returned by the jobs endpoint result in the BuildOperator return an error.

Continue wait is handled here for QueryOperator:
https://github.com/cube-js/airflow-provider-cube/blob/main/cube_provider/operators/cube.py#L117

But not here for build operator:
https://github.com/cube-js/airflow-provider-cube/blob/main/cube_provider/operators/cube.py#L193-L195