mwhitaker / dbt-action Goto Github PK
View Code? Open in Web Editor NEWRun and schedule dbt commands using Github Actions
License: MIT License
Run and schedule dbt commands using Github Actions
License: MIT License
The pip install
command in the DOCKERFILE to install the dbt-spark
package cause an update of the dbt-core
package. This makes the selection of a given base image version non-functional.
The following are excerpts from the build of the container used in this action:
Build container for action use: '/home/runner/work/_actions/mwhitaker/dbt-action/master/Dockerfile'.
/usr/bin/docker build -t fa4e14:e38c1f6462d14f3cbf2d823e43d0ffdb -f "/home/runner/work/_actions/mwhitaker/dbt-action/master/Dockerfile" "/home/runner/work/_actions/mwhitaker/dbt-action/master"
Sending build context to Docker daemon 16.38kB
Step 1/6 : ARG DBT_VERSION=0.19.1
Step 2/6 : FROM fishtownanalytics/dbt:${DBT_VERSION}
0.19.1: Pulling from fishtownanalytics/dbt
...
Successfully installed Jinja2-2.11.3 PyHive-0.6.4 dbt-core-0.21.0 dbt-extractor-0.4.0 dbt-spark-0.21.0 future-0.18.2 hologram-0.0.14 mashumaro-2.5 packaging-20.9 pure-sasl-0.6.2 pyparsing-2.4.7 sasl-0.3.1 sqlparams-3.0.0 thrift-0.11.0 thrift-sasl-0.4.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dbt 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
dbt-snowflake 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
dbt-redshift 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
dbt-postgres 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
dbt-bigquery 0.19.1 requires dbt-core==0.19.1, but you have dbt-core 0.21.0 which is incompatible.
...
I think this can be resolved by fixing the version of the additional dbt plugins installed to the same version specified by the DBT_VERSION
argument
i believe because of the need to have a colon the yaml isnt playing nicely, unless im missing something
example:
- name: dbt-deps
uses: mwhitaker/[email protected]
with:
dbt_command: "dbt deps --target pr --vars 'schema_id: 123' --profiles-dir ."
i consistently get an error like this no matter which syntax for vars i try:
Error: Invalid value for '--vars': String ''schema_id:' is not valid YAML
I also tried some other fixes like prefacing the dbt_command string with a | or a > or escaping the quotes, but i think the problem is the colon
if you have any other suggested fixes for this that would be awesome!
I have a strange error with the dbt action:
I wanted to set up dbt-action to just validate the source code so I set up the workflow like this:
name: CI
on:
push:
branches:
- main
- stage
- prod
jobs:
build:
name: Parse project
runs-on: ubuntu-latest
steps:
- name: dbt action
uses: mwhitaker/[email protected]
with:
dbt_command: dbt parse
But it fails with that it cannot find the dbt_project.yml
file:
But the dbt_project.yml
file is in the root path of the git repository:
Am I doing something wrong here?
When the action is run the following warning is shown:
Warning: The set-env
command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
Thanks for a useful action.
I haven't figured out how to run several dbt commands in series, such as dbt deps
followed by dbt build
. Is this possible with the current code?
Got a weird one here. I'm using the latest commit to master in my Github Action. I have a password which includes two '&' (ampersand) but I get the error:
trying to use user/password
saving console output in "./dbt_console_output.txt"
15:28:26 Encountered an error:
Runtime Error
dbt encountered an error while trying to read your profiles.yml file.
Runtime Error
Syntax error near line 9
------------------------------
6 | type: snowflake
7 | account:
8 | user: ***
9 | password: *SOMEOFMYPASSWORD_password_SOMEOFMYPASSWORD_password_***
10 | role: ANALYST
[11](https://github.com...) | database: FOO
[12](https://github.com...) | warehouse: BAR
Raw Error:
------------------------------
while scanning an alias
in "<unicode string>", line 9, column [17](https://github.com...)
did not find expected alphabetic or numeric character
in "<unicode string>", line 9, column [18](https://github.com...)
Where I've written SOMEOFMYPASSWORD....well, the logs are actually printing my password up to the amersands. Then it continues to print more of my password in plaintext until the second ampersand.
I have configured my password as per the Readme:
# profiles.yml
my_dataset:
outputs:
dev:
type: postgres
host: localhost
user: _user_ # this will be substituted during build time
pass: _password_ # this will be substituted during build time
port: 5432
dbname: tutorial
schema: dbt_tutorial
target: dev
And added the required secrets to allow the action to parse the authentication details.
Presumably this isn't expected behaviour?
The readme shows:
uses: mwhitaker/dbt-actions@master # -actions
when it should actually be:
uses: mwhitaker/dbt-action@master # -action
action
Hi, it would be awesome if newer versions of dbt(dbt-core) were supported. There have been some bug fixes in versions 1.0.1 in particular that I find very helpful
Running the action returns a warning in the logs:
bq. Warning: The set-output
command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
According to the blog item, set-output will throw an error starting 1st June 2023. This should be fixed latest some days in advance.
I gues its just these two code lines which needs to be changed:
Hi,
I started using this action in my workflow and it is great. I noticed that the "Build mwhitaker/dbt-action@master" step occurs every run, and GitHub must download all the layers, build the image, etc, even though nothing has changed with this image since the last run.
Is there a way to cache the built image so this step is much quicker? I was reading up on this but not sure if it would work for this scenario, and is probably something you'd have to add to the action itself: https://docs.docker.com/build/cache/backends/gha/
Thanks!
Instead of building Docker Images and manually / hosting on Docker Hub, couldn't we change update this project to reference dbt's hosted images on GitHub? See here.
This was released mid-December at Coalesce.
Can you release a new version? How should we specify that?
I have some private repositories as dependend packages for my dbt project as described in the dbt documentation here.
I need a way to add a private SSH key for private repositories via an environment variable.
Add a default ~/.ssh/config
:
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/id
Add the following commands to the dockerfile:
RUN echo "$DBT_ACTION_GIT_SSH_KEY" > ~/.ssh/id
RUN chmod 0600 ~/.ssh/id
RUN ssh-keyscan -H github.com >> ~/.ssh/known_hosts
*not tested
DataBricks dbt profiles use a token instead of username/password
Is that possible today?
The docker image does not connect properly to the Spark thrift server when the Spark thrift connection method is selected.
Error message:
root@a6a2937bc236:/github/workspace# dbt debug --profiles-dir profile
10:08:55 Running with dbt=1.1.0
dbt version: 1.1.0
python version: 3.9.9
python path: /usr/local/bin/python
os info: Linux-5.4.172-90.336.amzn2.x86_64-x86_64-with-glibc2.31
Using profiles.yml file at /github/workspace/profile/profiles.yml
Using dbt_project.yml file at /github/workspace/dbt_project.yml
Configuration:
profiles.yml file [OK found and valid]
dbt_project.yml file [OK found and valid]
Required dependencies:
- git [OK found]
Connection:
host: your-host
port: 10001
cluster: None
endpoint: None
schema: ethereum
organization: 0
Connection test: [ERROR]
1 check failed:
dbt was unable to connect to the specified database.
The database returned the following error:
>Runtime Error
Database Error
failed to connect
Check your database credentials and try again. For more information, visit:
https://docs.getdbt.com/docs/configure-your-profile
How to resolve it:
$ apt-get install -y libsasl2-modules
I'm probably doing something silly, but...
I have gotten my project working with dbt 0.21:
Running with dbt=0.21.0
However, it fails when I run the dba-action, because it is trying to use v0.19.0
Is this something I can configure? How would I do that?
Thanks!
P.S. Thanks for mainstreaming the DataBricks support. :-)
Hello,
I'm running into an issue where I want to use this dbt-action to deploy on a AWS Managed Workflow for Apache Airflow instance (AWS MWAA) running on airflow 2.4.3.
However, this 2.4.3 environment requires a dbt-core version greater than 1.3.0 as one its dependency (colorama==0.4.6) is conflicting with MWAA constraint as per below :
The conflict is caused by:
dbt-core 1.3.0 depends on colorama<0.4.6 and >=0.3.9
The user requested (constraint) colorama==0.4.6
Ideally, what would solve the issue is to have dbt-action able to run on the latest dbt-core version.
I'm also looking for any other kind of fix.
Thanks a lot !
Is there a way to use dbt actions with impersonated service account?
https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-impersonation
Hi thank you for maintaining this. To improve security I have the following suggestion:
Allow the authentication of the action via https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication-from-github-actions?hl=en
If this is already possible can you add documentation how to approach this?
Any guidance on how one could use this with third party adaptors like dbt-athena
? I take it that might require adding to the action's image build?
Hi, is there a plan to upgrade the dbt cli to version 1.5.0?
I have the following step
- name: dbt-run
uses: mwhitaker/[email protected]
with:
dbt_command: "dbt run --profiles-dir ."
env:
DBT_BIGQUERY_TOKEN: ${{ secrets.DBT_BIGQUERY_TOKEN }}
And I have a profiles.yml in the directory, and a secret value set in the repository. But, when that step runs, I get
ERROR: Database Error
[Errno 2] No such file or directory: './creds.json'
I thought that the creds.json file would be automatically created for me, so I'm unsure why it's failing.
It would be a great addition to include _ host _ as a variable in profiles.yml
, so that the host name can be added to Github's Action secrets as well.
Hello, tried adding github actions for DBT to our project, but got the following error.
trying` to parse bigquery token
success parsing plain token
saving console output in "harmonic/dbt_console_output.txt"
Running with dbt=0.21.0
Encountered an error while reading the project:
ERROR: Runtime Error
at path []: Additional properties are not allowed ('seed-paths', 'model-paths' were unexpected)
Error encountered in /github/workspace/harmonic/dbt_project.yml
Encountered an error:
Runtime Error
Could not run dbt
Our project yml based on DBT suggestion:
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
Is there a way to make this work without changing/removing 'seed-paths', 'model-paths'
?
Packages are now hosted here: https://github.com/orgs/dbt-labs/packages?visibility=public
Our team noticed that this wasn't working the same as it was before 2/25/2022. Here the error we're getting:
Need to get 456 kB of archives.
After this operation, 925 kB of additional disk space will be used.
Do you want to continue? [Y/n] Abort.
The command '/bin/sh -c apt-get update && apt-get install libsasl2-dev' returned a non-zero code: 1
Warning: Docker build failed with exit code 1, back off 1.525 seconds before retry.
/usr/bin/docker build -t 7db2d7:bdc4fc3503c14580a6da3d090a4fbc21 -f "/home/runner/work/_actions/mwhitaker/dbt-action/master/Dockerfile" "/home/runner/work/_actions/mwhitaker/dbt-action/master"
Sending build context to Docker daemon 16.9kB
Step 1/7 : ARG DBT_VERSION=1.0.0
Step 2/7 : FROM fishtownanalytics/dbt:${DBT_VERSION}
---> d874bacd7d19
Step 3/7 : RUN apt-get update && apt-get install libsasl2-dev
---> Running in c8ebff6b580f
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:2 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [121 kB]
Get:3 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:4 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2596 B]
Fetched 8506 kB in 1s (6234 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libsasl2-2 libsasl2-modules-db
Recommended packages:
libsasl2-modules
The following NEW packages will be installed:
libsasl2-dev
The following packages will be upgraded:
libsasl2-2 libsasl2-modules-db
2 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 456 kB of archives.
After this operation, 925 kB of additional disk space will be used.
Do you want to continue? [Y/n] Abort.
The command '/bin/sh -c apt-get update && apt-get install libsasl2-dev' returned a non-zero code: 1
Warning: Docker build failed with exit code 1, back off 4.486 seconds before retry.
/usr/bin/docker build -t 7db2d7:bdc4fc3503c14580a6da3d090a4fbc21 -f "/home/runner/work/_actions/mwhitaker/dbt-action/master/Dockerfile" "/home/runner/work/_actions/mwhitaker/dbt-action/master"
Sending build context to Docker daemon 16.9kB
Step 1/7 : ARG DBT_VERSION=1.0.0
Step 2/7 : FROM fishtownanalytics/dbt:${DBT_VERSION}
---> d874bacd7d19
Step 3/7 : RUN apt-get update && apt-get install libsasl2-dev
---> Running in a97435e97b3e
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:2 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:4 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [121 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2596 B]
Fetched 8506 kB in 1s (6614 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libsasl2-2 libsasl2-modules-db
Recommended packages:
libsasl2-modules
The following NEW packages will be installed:
libsasl2-dev
The following packages will be upgraded:
libsasl2-2 libsasl2-modules-db
2 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 456 kB of archives.
After this operation, 925 kB of additional disk space will be used.
Do you want to continue? [Y/n] Abort.
The command '/bin/sh -c apt-get update && apt-get install libsasl2-dev' returned a non-zero code: 1
Error: Docker build failed with exit code 1
Hi, I'm trying to set this up to work with Snowflake, but struggling to get the credentials passed to the profiles.yml file.
I've set up the profile with _user_
and _password_
, and created env vars in the workflow file, which map back to my stored secrets.
docs:
account: <my_account>
database: <my_db>
user: _user_
password: _password_
role: <my_role>
schema: dev
threads: 2
type: snowflake
warehouse: <my_compute>
But this fails to authenticate. If I hard-code the credentials in the profiles file, it works fine. If I hard-code the environment variables (rather than grabbing from secrets) it fails . But if I print the env_vars to console, they're there - so I think it's failing to map the environment variables to _user_
and _password_
.
Have I misunderstood how to configure this, or is this a bug?
My workflow step looks like this...
- name: Generate DBT Docs
uses: mwhitaker/dbt-action@master
with:
dbt_command: "dbt docs generate --profiles-dir ../docs_hosting/"
dbt_project_folder: "dbt/dbtlearn"
env:
DBT_USER: ${{ secrets.SF_USER }}
DBT_PASSWORD: ${{ secrets.SF_PWD }}
With the latest release, actions have begun to fail due to misconfiguration of logging.
dbt project folder set as: "/github/workspace/data_marts"
trying to parse bigquery token
success parsing plain token
saving console output in "/github/workspace/data_marts/dbt.log"
Error: 10:23:[20](https://github.com/**********/actions/runs/**********/jobs/**********#step:6:21).255872 [error] [MainThread]: Encountered an error:
[Errno 20] Not a directory: '/github/workspace/data_marts/dbt.log/dbt.log'
10:23:20 Encountered an error:
[Errno 20] Not a directory: '/github/workspace/data_marts/dbt.log/dbt.log'
Error: 10:[23](https://github.com/**********/actions/runs/**********/jobs/**********#step:6:24):20.[25](https://github.com/**********/actions/runs/**********/jobs/**********#step:6:26)7841 [error] [MainThread]: Traceback (most recent call last):
It seems that dbt-core expects that the new DBT_LOG_PATH
env var (also available as the log-path yml config) points to a directory. However, the DBT_LOG_PATH
in dbt-action seems to serve different purpose at points at the log file itself..
Perhaps the DBT_LOG_PATH
env var in dbt-action is no longer necessary given the new dbt-core log-path config option? Happy to submit a PR if you are accepting them right now! 🙂
I would like to use dbt version 1.6.1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.