jayamanikharyono / airflow-dag-action Goto Github PK
View Code? Open in Web Editor NEWGitHub Actions to Validate DAGs, Variables and Dependencies upon Pull Request
License: MIT License
GitHub Actions to Validate DAGs, Variables and Dependencies upon Pull Request
License: MIT License
tests/dags/shared_var.py
contains CONSTANTS that is used by multiple dags within the dag root AIRFLOW__CORE__DAGS_FOLDER
, but dag-action fails
dag_validation.py:25: AssertionError
------------------------------ Captured log call -------------------------------
INFO root:dag_validation.py:18 DAGs dir : tests/dags
INFO airflow.models.dagbag.DagBag:dagbag.py:500 Filling up the DagBag from tests/dags
ERROR airflow.models.dagbag.DagBag:dagbag.py:334 Failed to import: tests/dags/test_dag.py
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 331, in _load_modules_from_file
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "tests/dags/test_dag.py", line 8, in <module>
from shared_var import image
ModuleNotFoundError: No module named 'shared_var'
A new issue occurs when trigger action from a forked repo in this PR caused the GitHub Actions token used with degraded permission from "write/read" to "read-only".
1 of 1 variables successfully updated.
Pytest exited 1
Traceback (most recent call last):
File "alert.py", line 37, in <module>
comment_pr(args.repo_token, args.log_filename)
File "alert.py", line 25, in comment_pr
pr.create_issue_comment("```" + message + "```")
File "/usr/local/lib/python3.7/site-packages/github/PullRequest.py", line 457, in create_issue_comment
"POST", f"{self.issue_url}/comments", input=post_parameters
File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 355, in requestJsonAndCheck
verb, url, parameters, headers, input, self.__customConnection(url)
File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 378, in __check
raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 403 {"message": "Resource not accessible by integration", "documentation_url": "https://docs.github.com/rest/reference/issues#create-an-issue-comment"}
Pytest did not exit 0
Hi there @jayamanikharyono ,
I'm a colleague of kieronellis, a contributor to this repository. I have a request ๐ :
Would be great if you could update the python version in the dockerfile of this repository to 3.8. We utilise this repo for validating airflow DAGs, but upon updating a package requirement which requires python 3.8, the github action no longer is able to build because of that. We are anyhow running our production environment of airflow via python 3.8
I do not have write access to this repo, however I have made a fork with an example - linked here - and copied below. This builds successfully when using a requirements file of packages requiring python>=3.8, for example, airflow-provider-fivetran-async
##dockerfile:
FROM python:3.8
RUN python -m venv /opt/venv
# Install airflow
ENV PYTHON_VERSION 3.8
ENV AIRFLOW_VERSION=2.2.4
RUN pip install --upgrade pip
ENV CONSTRAINT_URL "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
RUN pip install "apache-airflow[async,postgres,google,cncf.kubernetes]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
# Install Deps
RUN pip install google-cloud-storage
RUN pip install google-auth-httplib2
RUN pip install google-api-python-client
RUN pip install pandas-gbq
RUN pip install pytest
RUN pip install PyGithub==1.55
RUN mkdir /action
COPY dag_validation.py /action/dag_validation.py
COPY alert.py /action/alert.py
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
@jayamanikharyono
(Duplicate of this comment: #32 (comment))
Request for a new release (v2.5) of this repo to be made, so that recent updates can be included and it can be referenced properly in external workflows, i.e.:
uses: jayamanikharyono/[email protected]
Many thanks! ๐
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.