jayamanikharyono / airflow-dag-action

GitHub Actions to Validate DAGs, Variables and Dependencies upon Pull Request

Hi there 👋

---

👩‍💻 About Me :

I am a Data Engineer from Indonesia.

• 🔭 I’m working as a Data Engineer @Tiket.com and contributing (via words and codes) to data engineering related things.

• 🌱 Exploring Technical Content Writing and Data Engineering Related Tech.

• ⚡ In my free time, I solve problems on GeeksforGeeks, read tech articles and build things.

• 👯 I’m looking to collaborate on any open source project, expecially on projects that can help fellows DE around the Globe.

• 📫 How to reach me :

---

🔥 My Stats :

---

✍️ Blog Posts :

• A simple way to Validate your Airflow DAGs on Pull Request
• CICD your Tableau Dashboard with Github Actions
• How I do DBT (data build tools)

airflow-dag-action's Issues

Resource not accessible by integration

A new issue occurs when trigger action from a forked repo in this PR caused the GitHub Actions token used with degraded permission from "write/read" to "read-only".

1 of 1 variables successfully updated.
Pytest exited 1
Traceback (most recent call last):
  File "alert.py", line 37, in <module>
    comment_pr(args.repo_token, args.log_filename)
  File "alert.py", line 25, in comment_pr
    pr.create_issue_comment("```" + message + "```")
  File "/usr/local/lib/python3.7/site-packages/github/PullRequest.py", line 457, in create_issue_comment
    "POST", f"{self.issue_url}/comments", input=post_parameters
  File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 355, in requestJsonAndCheck
    verb, url, parameters, headers, input, self.__customConnection(url)
  File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 378, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 403 {"message": "Resource not accessible by integration", "documentation_url": "https://docs.github.com/rest/reference/issues#create-an-issue-comment"}
Pytest did not exit 0

dag action fails trying to import from another file in dag folder

tests/dags/shared_var.py contains CONSTANTS that is used by multiple dags within the dag root AIRFLOW__CORE__DAGS_FOLDER, but dag-action fails

dag_validation.py:25: AssertionError
------------------------------ Captured log call -------------------------------
INFO     root:dag_validation.py:18 DAGs dir : tests/dags
INFO     airflow.models.dagbag.DagBag:dagbag.py:500 Filling up the DagBag from tests/dags
ERROR    airflow.models.dagbag.DagBag:dagbag.py:334 Failed to import: tests/dags/test_dag.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 331, in _load_modules_from_file
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "tests/dags/test_dag.py", line 8, in <module>
    from shared_var import image
ModuleNotFoundError: No module named 'shared_var'

Update python version in dockerfile to 3.8?

Hi there @jayamanikharyono ,

I'm a colleague of kieronellis, a contributor to this repository. I have a request 🙏 :

Would be great if you could update the python version in the dockerfile of this repository to 3.8. We utilise this repo for validating airflow DAGs, but upon updating a package requirement which requires python 3.8, the github action no longer is able to build because of that. We are anyhow running our production environment of airflow via python 3.8

I do not have write access to this repo, however I have made a fork with an example - linked here - and copied below. This builds successfully when using a requirements file of packages requiring python>=3.8, for example, airflow-provider-fivetran-async

##dockerfile:

FROM python:3.8

RUN python -m venv /opt/venv

# Install airflow
ENV PYTHON_VERSION 3.8
ENV AIRFLOW_VERSION=2.2.4
RUN pip install --upgrade pip
ENV CONSTRAINT_URL "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
RUN pip install "apache-airflow[async,postgres,google,cncf.kubernetes]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

# Install Deps
RUN pip install google-cloud-storage
RUN pip install google-auth-httplib2
RUN pip install google-api-python-client
RUN pip install pandas-gbq
RUN pip install pytest
RUN pip install PyGithub==1.55

RUN mkdir /action
COPY dag_validation.py /action/dag_validation.py
COPY alert.py /action/alert.py

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

Request for new release

@jayamanikharyono
(Duplicate of this comment: #32 (comment))

Request for a new release (v2.5) of this repo to be made, so that recent updates can be included and it can be referenced properly in external workflows, i.e.:

uses: jayamanikharyono/[email protected]

Many thanks! 🙏