Coder Social home page Coder Social logo

jayamanikharyono / airflow-dag-action Goto Github PK

View Code? Open in Web Editor NEW
19.0 19.0 14.0 1.92 MB

GitHub Actions to Validate DAGs, Variables and Dependencies upon Pull Request

License: MIT License

Dockerfile 11.06% Shell 14.80% Python 74.14%
airflow cicd github-actions python

airflow-dag-action's Issues

dag action fails trying to import from another file in dag folder

tests/dags/shared_var.py contains CONSTANTS that is used by multiple dags within the dag root AIRFLOW__CORE__DAGS_FOLDER, but dag-action fails

dag_validation.py:25: AssertionError
------------------------------ Captured log call -------------------------------
INFO     root:dag_validation.py:18 DAGs dir : tests/dags
INFO     airflow.models.dagbag.DagBag:dagbag.py:500 Filling up the DagBag from tests/dags
ERROR    airflow.models.dagbag.DagBag:dagbag.py:334 Failed to import: tests/dags/test_dag.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 331, in _load_modules_from_file
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "tests/dags/test_dag.py", line 8, in <module>
    from shared_var import image
ModuleNotFoundError: No module named 'shared_var'

Resource not accessible by integration

A new issue occurs when trigger action from a forked repo in this PR caused the GitHub Actions token used with degraded permission from "write/read" to "read-only".

1 of 1 variables successfully updated.
Pytest exited 1
Traceback (most recent call last):
  File "alert.py", line 37, in <module>
    comment_pr(args.repo_token, args.log_filename)
  File "alert.py", line 25, in comment_pr
    pr.create_issue_comment("```" + message + "```")
  File "/usr/local/lib/python3.7/site-packages/github/PullRequest.py", line 457, in create_issue_comment
    "POST", f"{self.issue_url}/comments", input=post_parameters
  File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 355, in requestJsonAndCheck
    verb, url, parameters, headers, input, self.__customConnection(url)
  File "/usr/local/lib/python3.7/site-packages/github/Requester.py", line 378, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 403 {"message": "Resource not accessible by integration", "documentation_url": "https://docs.github.com/rest/reference/issues#create-an-issue-comment"}
Pytest did not exit 0

Update python version in dockerfile to 3.8?

Hi there @jayamanikharyono ,

I'm a colleague of kieronellis, a contributor to this repository. I have a request ๐Ÿ™ :

Would be great if you could update the python version in the dockerfile of this repository to 3.8. We utilise this repo for validating airflow DAGs, but upon updating a package requirement which requires python 3.8, the github action no longer is able to build because of that. We are anyhow running our production environment of airflow via python 3.8

I do not have write access to this repo, however I have made a fork with an example - linked here - and copied below. This builds successfully when using a requirements file of packages requiring python>=3.8, for example, airflow-provider-fivetran-async

##dockerfile:

FROM python:3.8

RUN python -m venv /opt/venv

# Install airflow
ENV PYTHON_VERSION 3.8
ENV AIRFLOW_VERSION=2.2.4
RUN pip install --upgrade pip
ENV CONSTRAINT_URL "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
RUN pip install "apache-airflow[async,postgres,google,cncf.kubernetes]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

# Install Deps
RUN pip install google-cloud-storage
RUN pip install google-auth-httplib2
RUN pip install google-api-python-client
RUN pip install pandas-gbq
RUN pip install pytest
RUN pip install PyGithub==1.55

RUN mkdir /action
COPY dag_validation.py /action/dag_validation.py
COPY alert.py /action/alert.py

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.