gerrykou / airflow_energy

Get energy data from api, monitoring with airflow

Readme

An app to monitor the task of downloading data to files from an API using Airflow

Installing / Getting Started

Follow instructions to install docker compose and set up the right Airflow user
https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#running-airflow-in-docker

Initial Configuration

Inside the dags folder, add a config_local.py file that should contain the url and credentials for the API

ENERGY_API_URL = 'your_url'
ENERGY_API_UUID = 'your_uuid'
ENERGY_API_TOKEN = 'your_token'

Building

docker compose up airflow-init
docker compose up

run docker ps to find airflow-scheduler container id

docker ps 
docker exec -it container-id bash

https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html#backfill

airflow dags backfill --start-date START_DATE --end-date END_DATE dag_id   

e.g

airflow dags backfill --start-date 2022-01-19 --end-date 2022-01-20 my_dag

docker-compose down

Visit localhost
http://localhost:8080/

Readme template from
https://github.com/jehna/readme-best-practices/blob/master/README-default.md

docker exec -it 7782a422355a bash