Comments (5)
Hey @jacobhjkim, just curious, what would be your use-case for stateful KEDA workers?
We also just merged a slightly more open-source friendly version of this chart into the official Apache Airflow repository, so a PR over on the main Airflow repo would be great! It lives under the chart
subdirectory.
from airflow-chart.
I currently have an issue with webserver not being able to fetch logs from each worker. I thought that this issue would be solved by using StatefulSet and assigning a fixed DNS name to each individual worker.
It's great to hear that this chart repo has been merged into the official Apache Airflow repo!
from airflow-chart.
Oh, that's a good point. Even with a StatefulSet
, if you attempt to view the logs from a worker that has been scaled away, you'd get he same error.
I think the only way to get around this would be to use a centralized logging solution, like one of the object store backends (s3, gcs, wasb) or elastic.
from airflow-chart.
@jacobhjkim That's an interesting point re: logs. How would this work if a worker is scaled down? Would a worker need to spin up again to use that volume to pull the logs?=
from airflow-chart.
Right, using StatefulSet
won't solve the issue mentioned above. Will close this issue. (Sorry for the late reply 😅)
from airflow-chart.
Related Issues (20)
- Spark binaries HOT 1
- Default ImagePullPolicy
- No gitSync config in values.yml HOT 1
- Astronomer maintenance of this repo? HOT 3
- Please update airflow version to 1.10.14 by default HOT 1
- If use 1.10.14-buster error alembic.script.revision.ResolutionError: No such revision or branch 'e959f08ac86c' HOT 1
- Can not change webserver.defaultUser.password when install by helm HOT 1
- Use helm-doc to generate parameter table
- Using Google Cloud Storage remote logging HOT 1
- Invalid image name when the image tag is integer HOT 6
- configmap sourced env values
- fernetKey when using airflow as dependency
- airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'serve_logs' HOT 2
- Any helm upgrade with KEDA errors the scheduler HOT 1
- Syncing dags from multiple git-repos HOT 3
- Add imagePullSecret as a parameter to pull images from Private Container Registries HOT 1
- Help needed - additionalVolume not parsable by helm with version 0.25.8 HOT 1
- Unable to install chart "Deployment is not ready: airflow/airflow-scheduler" HOT 2
- Getting Airflow 404 = Lots of circles Error HOT 3
- Log Grooming script envvars HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from airflow-chart.