mwhitaker / dbt-action Goto Github PK

The pip install command in the DOCKERFILE to install the dbt-spark package cause an update of the dbt-core package. This makes the selection of a given base image version non-functional.

The following are excerpts from the build of the container used in this action:

Build container for action use: '/home/runner/work/_actions/mwhitaker/dbt-action/master/Dockerfile'.
  /usr/bin/docker build -t fa4e14:e38c1f6462d14f3cbf2d823e43d0ffdb -f "/home/runner/work/_actions/mwhitaker/dbt-action/master/Dockerfile" "/home/runner/work/_actions/mwhitaker/dbt-action/master"
  Sending build context to Docker daemon  16.38kB
  
  Step 1/6 : ARG DBT_VERSION=0.19.1
  Step 2/6 : FROM fishtownanalytics/dbt:${DBT_VERSION}
  0.19.1: Pulling from fishtownanalytics/dbt

...

  Successfully installed Jinja2-2.11.3 PyHive-0.6.4 dbt-core-0.21.0 dbt-extractor-0.4.0 dbt-spark-0.21.0 future-0.18.2 hologram-0.0.14 mashumaro-2.5 packaging-20.9 pure-sasl-0.6.2 pyparsing-2.4.7 sasl-0.3.1 sqlparams-3.0.0 thrift-0.11.0 thrift-sasl-0.4.3
  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
  dbt 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
  dbt-snowflake 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
  dbt-redshift 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
  dbt-postgres 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
  dbt-bigquery 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
...

I think this can be resolved by fixing the version of the additional dbt plugins installed to the same version specified by the DBT_VERSION argument

i believe because of the need to have a colon the yaml isnt playing nicely, unless im missing something

example:

    - name: dbt-deps
      uses: mwhitaker/[email protected]
      with:
        dbt_command: "dbt deps --target pr --vars 'schema_id: 123' --profiles-dir ."

i consistently get an error like this no matter which syntax for vars i try:

Error: Invalid value for '--vars': String ''schema_id:' is not valid YAML

I also tried some other fixes like prefacing the dbt_command string with a | or a > or escaping the quotes, but i think the problem is the colon

if you have any other suggested fixes for this that would be awesome!

I have a strange error with the dbt action:

I wanted to set up dbt-action to just validate the source code so I set up the workflow like this:

name: CI

on:
  push:
    branches:
      - main
      - stage
      - prod

jobs:
  build:
    name: Parse project
    runs-on: ubuntu-latest
    steps:
      - name: dbt action
        uses: mwhitaker/[email protected]
        with:
          dbt_command: dbt parse

But it fails with that it cannot find the dbt_project.yml file:

But the dbt_project.yml file is in the root path of the git repository:

Am I doing something wrong here?

When the action is run the following warning is shown:

Warning: The set-env command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/

Thanks for a useful action.
I haven't figured out how to run several dbt commands in series, such as dbt deps followed by dbt build. Is this possible with the current code?

Got a weird one here. I'm using the latest commit to master in my Github Action. I have a password which includes two '&' (ampersand) but I get the error:

trying to use user/password
saving console output in "./dbt_console_output.txt"
15:28:26  Encountered an error:
Runtime Error
  
  dbt encountered an error while trying to read your profiles.yml file.
  
  Runtime Error
    Syntax error near line 9
    ------------------------------
    6  |       type: snowflake
    7  |       account:
    8  |       user: ***
    9  |       password: *SOMEOFMYPASSWORD_password_SOMEOFMYPASSWORD_password_***
    10 |       role: ANALYST
    [11](https://github.com...) |       database: FOO
    [12](https://github.com...) |       warehouse: BAR
    
    Raw Error:
    ------------------------------
    while scanning an alias
      in "<unicode string>", line 9, column [17](https://github.com...)
    did not find expected alphabetic or numeric character
      in "<unicode string>", line 9, column [18](https://github.com...)

Where I've written SOMEOFMYPASSWORD....well, the logs are actually printing my password up to the amersands. Then it continues to print more of my password in plaintext until the second ampersand.

I have configured my password as per the Readme:

# profiles.yml
my_dataset:
  outputs:
    dev:
      type: postgres
      host: localhost
      user: _user_      # this will be substituted during build time
      pass: _password_  # this will be substituted during build time
      port: 5432
      dbname: tutorial
      schema: dbt_tutorial
  target: dev

And added the required secrets to allow the action to parse the authentication details.

Presumably this isn't expected behaviour?

The readme shows:

 uses: mwhitaker/dbt-actions@master # -actions

when it should actually be:

uses: mwhitaker/dbt-action@master # -action

action

Hi, it would be awesome if newer versions of dbt(dbt-core) were supported. There have been some bug fixes in versions 1.0.1 in particular that I find very helpful

Running the action returns a warning in the logs:

bq. Warning: The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

According to the blog item, set-output will throw an error starting 1st June 2023. This should be fixed latest some days in advance.

I gues its just these two code lines which needs to be changed:

• /entrypoint.sh#L43
• /entrypoint.sh#L47

Hi,

I started using this action in my workflow and it is great. I noticed that the "Build mwhitaker/dbt-action@master" step occurs every run, and GitHub must download all the layers, build the image, etc, even though nothing has changed with this image since the last run.

Is there a way to cache the built image so this step is much quicker? I was reading up on this but not sure if it would work for this scenario, and is probably something you'd have to add to the action itself: https://docs.docker.com/build/cache/backends/gha/

Thanks!

Instead of building Docker Images and manually / hosting on Docker Hub, couldn't we change update this project to reference dbt's hosted images on GitHub? See here.

This was released mid-December at Coalesce.
Can you release a new version? How should we specify that?

https://coalesce.getdbt.com/talks/dbt-v10-reveal/

I have some private repositories as dependend packages for my dbt project as described in the dbt documentation here.

I need a way to add a private SSH key for private repositories via an environment variable.

Sample solution:

Add a default ~/.ssh/config:

Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/id

Add the following commands to the dockerfile:

RUN echo "$DBT_ACTION_GIT_SSH_KEY" > ~/.ssh/id
RUN chmod 0600 ~/.ssh/id
RUN ssh-keyscan -H github.com >> ~/.ssh/known_hosts

*not tested

DataBricks dbt profiles use a token instead of username/password

Is that possible today?

The docker image does not connect properly to the Spark thrift server when the Spark thrift connection method is selected.

Error message:

root@a6a2937bc236:/github/workspace# dbt debug --profiles-dir profile
10:08:55  Running with dbt=1.1.0
dbt version: 1.1.0
python version: 3.9.9
python path: /usr/local/bin/python
os info: Linux-5.4.172-90.336.amzn2.x86_64-x86_64-with-glibc2.31
Using profiles.yml file at /github/workspace/profile/profiles.yml
Using dbt_project.yml file at /github/workspace/dbt_project.yml

Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

Connection:
  host: your-host
  port: 10001
  cluster: None
  endpoint: None
  schema: ethereum
  organization: 0
  Connection test: [ERROR]

1 check failed:
dbt was unable to connect to the specified database.
The database returned the following error:

  >Runtime Error
  Database Error
    failed to connect

Check your database credentials and try again. For more information, visit:
https://docs.getdbt.com/docs/configure-your-profile

ref：Can't connect to unsecured hive. SASL error: TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2 · Issue #161 · dropbox/PyHive (github.com)

How to resolve it:

$ apt-get install -y libsasl2-modules

I'm probably doing something silly, but...

I have gotten my project working with dbt 0.21:

Running with dbt=0.21.0

However, it fails when I run the dba-action, because it is trying to use v0.19.0

Is this something I can configure? How would I do that?

Thanks!
P.S. Thanks for mainstreaming the DataBricks support. :-)

Hello,

I'm running into an issue where I want to use this dbt-action to deploy on a AWS Managed Workflow for Apache Airflow instance (AWS MWAA) running on airflow 2.4.3.

However, this 2.4.3 environment requires a dbt-core version greater than 1.3.0 as one its dependency (colorama==0.4.6) is conflicting with MWAA constraint as per below :

The conflict is caused by:
    dbt-core 1.3.0 depends on colorama<0.4.6 and >=0.3.9
    The user requested (constraint) colorama==0.4.6

Ideally, what would solve the issue is to have dbt-action able to run on the latest dbt-core version.

I'm also looking for any other kind of fix.

Thanks a lot !

Is there a way to use dbt actions with impersonated service account?

https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-impersonation

Hi thank you for maintaining this. To improve security I have the following suggestion:

Allow the authentication of the action via https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication-from-github-actions?hl=en

If this is already possible can you add documentation how to approach this?

Any guidance on how one could use this with third party adaptors like dbt-athena ? I take it that might require adding to the action's image build?

Hi, is there a plan to upgrade the dbt cli to version 1.5.0?

Unfortunately, the latest v1.3.0 version points to a wrong commit. Screenshots -->

I have the following step

- name: dbt-run
  uses: mwhitaker/[email protected]
  with:
    dbt_command: "dbt run --profiles-dir ."
  env:
    DBT_BIGQUERY_TOKEN: ${{ secrets.DBT_BIGQUERY_TOKEN }}

And I have a profiles.yml in the directory, and a secret value set in the repository. But, when that step runs, I get

ERROR: Database Error
  [Errno 2] No such file or directory: './creds.json'

I thought that the creds.json file would be automatically created for me, so I'm unsure why it's failing.

It would be a great addition to include _ host _ as a variable in profiles.yml, so that the host name can be added to Github's Action secrets as well.

Hello, tried adding github actions for DBT to our project, but got the following error.

trying` to parse bigquery token
success parsing plain token
saving console output in "harmonic/dbt_console_output.txt"
Running with dbt=0.21.0
Encountered an error while reading the project:
  ERROR: Runtime Error
  at path []: Additional properties are not allowed ('seed-paths', 'model-paths' were unexpected)

Error encountered in /github/workspace/harmonic/dbt_project.yml
Encountered an error:
Runtime Error
  Could not run dbt

Our project yml based on DBT suggestion:

model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

Is there a way to make this work without changing/removing 'seed-paths', 'model-paths' ?

Packages are now hosted here: https://github.com/orgs/dbt-labs/packages?visibility=public

Our team noticed that this wasn't working the same as it was before 2/25/2022. Here the error we're getting:

Need to get 456 kB of archives.
After this operation, 925 kB of additional disk space will be used.
Do you want to continue? [Y/n] Abort.
The command '/bin/sh -c apt-get update && apt-get install libsasl2-dev' returned a non-zero code: 1
Warning: Docker build failed with exit code 1, back off 1.525 seconds before retry.
/usr/bin/docker build -t 7db2d7:bdc4fc3503c14580a6da3d090a4fbc21 -f "/home/runner/work/_actions/mwhitaker/dbt-action/master/Dockerfile" "/home/runner/work/_actions/mwhitaker/dbt-action/master"
Sending build context to Docker daemon 16.9kB

Step 1/7 : ARG DBT_VERSION=1.0.0
Step 2/7 : FROM fishtownanalytics/dbt:${DBT_VERSION}
---> d874bacd7d19
Step 3/7 : RUN apt-get update && apt-get install libsasl2-dev
---> Running in c8ebff6b580f
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:2 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [121 kB]
Get:3 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:4 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2596 B]
Fetched 8506 kB in 1s (6234 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libsasl2-2 libsasl2-modules-db
Recommended packages:
libsasl2-modules
The following NEW packages will be installed:
libsasl2-dev
The following packages will be upgraded:
libsasl2-2 libsasl2-modules-db
2 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 456 kB of archives.
After this operation, 925 kB of additional disk space will be used.
Do you want to continue? [Y/n] Abort.
The command '/bin/sh -c apt-get update && apt-get install libsasl2-dev' returned a non-zero code: 1
Warning: Docker build failed with exit code 1, back off 4.486 seconds before retry.
/usr/bin/docker build -t 7db2d7:bdc4fc3503c14580a6da3d090a4fbc21 -f "/home/runner/work/_actions/mwhitaker/dbt-action/master/Dockerfile" "/home/runner/work/_actions/mwhitaker/dbt-action/master"
Sending build context to Docker daemon 16.9kB

Step 1/7 : ARG DBT_VERSION=1.0.0
Step 2/7 : FROM fishtownanalytics/dbt:${DBT_VERSION}
---> d874bacd7d19
Step 3/7 : RUN apt-get update && apt-get install libsasl2-dev
---> Running in a97435e97b3e
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:2 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:4 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [121 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2596 B]
Fetched 8506 kB in 1s (6614 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libsasl2-2 libsasl2-modules-db
Recommended packages:
libsasl2-modules
The following NEW packages will be installed:
libsasl2-dev
The following packages will be upgraded:
libsasl2-2 libsasl2-modules-db
2 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 456 kB of archives.
After this operation, 925 kB of additional disk space will be used.
Do you want to continue? [Y/n] Abort.
The command '/bin/sh -c apt-get update && apt-get install libsasl2-dev' returned a non-zero code: 1
Error: Docker build failed with exit code 1

Hi, I'm trying to set this up to work with Snowflake, but struggling to get the credentials passed to the profiles.yml file.

I've set up the profile with _user_ and _password_ , and created env vars in the workflow file, which map back to my stored secrets.

docs:
 account: <my_account>
 database: <my_db>
 user: _user_
 password: _password_
 role: <my_role>
 schema: dev
 threads: 2
 type: snowflake
 warehouse: <my_compute>

But this fails to authenticate. If I hard-code the credentials in the profiles file, it works fine. If I hard-code the environment variables (rather than grabbing from secrets) it fails . But if I print the env_vars to console, they're there - so I think it's failing to map the environment variables to _user_ and _password_ .

Have I misunderstood how to configure this, or is this a bug?

My workflow step looks like this...

      - name: Generate DBT Docs
        uses: mwhitaker/dbt-action@master
        with:
          dbt_command: "dbt docs generate --profiles-dir ../docs_hosting/"
          dbt_project_folder: "dbt/dbtlearn"
        env:
          DBT_USER: ${{ secrets.SF_USER }}
          DBT_PASSWORD: ${{ secrets.SF_PWD }}

With the latest release, actions have begun to fail due to misconfiguration of logging.

dbt project folder set as: "/github/workspace/data_marts"
trying to parse bigquery token
success parsing plain token
saving console output in "/github/workspace/data_marts/dbt.log"
Error: 10:23:[20](https://github.com/**********/actions/runs/**********/jobs/**********#step:6:21).255872 [error] [MainThread]: Encountered an error:
[Errno 20] Not a directory: '/github/workspace/data_marts/dbt.log/dbt.log'
10:23:20  Encountered an error:
[Errno 20] Not a directory: '/github/workspace/data_marts/dbt.log/dbt.log'
Error: 10:[23](https://github.com/**********/actions/runs/**********/jobs/**********#step:6:24):20.[25](https://github.com/**********/actions/runs/**********/jobs/**********#step:6:26)7841 [error] [MainThread]: Traceback (most recent call last):

It seems that dbt-core expects that the new DBT_LOG_PATH env var (also available as the log-path yml config) points to a directory. However, the DBT_LOG_PATH in dbt-action seems to serve different purpose at points at the log file itself..

Perhaps the DBT_LOG_PATH env var in dbt-action is no longer necessary given the new dbt-core log-path config option? Happy to submit a PR if you are accepting them right now! 🙂

I would like to use dbt version 1.6.1

Could you add 'echo' stages or headers to identify what each of the different steps are doing?

It looks like these are deps, debug, run, and test. They often take a long time, so it would be somewhat comforting to know exactly where things stood, at a glance.