Coder Social home page Coder Social logo

apache_airflow_on_eks's People

Contributors

hubaym avatar nikvakl avatar tarpeti avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

apache_airflow_on_eks's Issues

terraform apply failing

I ran into an issue while trying out this repository.

the issue occurred during the "terraform apply" step.


Error: Provider produced inconsistent final plan

When expanding the plan for
module.eks_cluster.null_resource.wait_for_cluster[0] to include new values
learned so far during apply, provider "registry.terraform.io/hashicorp/null"
changed the planned action from CreateThenDelete to DeleteThenCreate.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

it happens when terraform tries to create the aws_launch_configuration for both worker.
Because of the error both neither of the workers are created.

These are the variables that I added:


variable "project_name" {
 description = "Name for the test project"
 default = "test-project"
 type = string
}


variable "environment" {
 default = "dev"
}

variable "aws_profile" {
 default = "default"
}

variable "aws_region" {
 default = "us-east-2"
}

// This will be the ip that will be allowed to connect to Airflow and to the monitoring services by default
variable "allowed_ip" {
 default = "0.0.0.0/0"
}

variable "nodes_instance_type_1" {
 default = "t3.medium"
 type = string
}

variable "nodes_instance_type_2" {
 default = "t3.large"
 type = string
}

// This RDS Postgres is used by Airflow to store metadata
variable "postgres_user" {
 default = "name"
 type = string
}

variable "postgres_password" {
 default = "XXXXXXX"
 type = string
}

And this is the terraform version I am using:


Terraform v0.13.2
+ provider registry.terraform.io/hashicorp/aws v3.5.0
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.1
+ provider registry.terraform.io/hashicorp/local v1.4.0
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/hashicorp/random v2.3.0
+ provider registry.terraform.io/hashicorp/template v2.1.2

Airflow depployment failing on EKS Fargate

I have my EKS Fargate cluster with no EC2 as node group. When deployed airflow in EKS using helm the deployment is failing and the pods are terminating and restarting continuously. But the same docker image when taken to EKS with Ec2 airflow is running without issues. Please help me on this.

Connect external repo as a dag folder

First off, thanks for the heavy lifting here!

Whenever there's any change to the Airflow Dags, the Dockerfile inside the dag folder can be built into an image, and pushed into the ECR repository.

Would it be possible to reference a git repository where our Airflow dags are located separately? As Airflow dags usually change more frequently it would be great to have less deployments.

Thanks in advance!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.