Comments (3)
The pip issue suggests switching to a requirements.in file and pip-tools to compile it pushing the dependency resolution step out of installation time. I did that with socorro and tecken a while back. We could do that here.
I don't know where the requirements file is coming from. It's this line in dataproc_init.sh
:
telemetry-airflow/dataproc_bootstrap/dataproc_init.sh
Lines 26 to 27 in 60f7dd1
Maybe that requirements file is already compiled in which case maybe all we need to do is update the requirements.in
and recompile it?
from telemetry-airflow.
I don't know where the requirements file is coming from. It's this line in dataproc_init.sh
It looks like it's defined within telemetry-airflow/dataproc_bootstrap and the contents of that folder are rsync'd to GCS.
Looks like it is not currently compiled, but we could do so within telemetry-airflow and commit the results.
from telemetry-airflow.
The pip issue suggests switching to a requirements.in file and pip-tools to compile it pushing the dependency resolution step out of installation time. I did that with socorro and tecken a while back. We could do that here.
I strongly agree with using pip-tools for compiling requirements. We use it for the airflow container to pin dependencies (#1008)
from telemetry-airflow.
Related Issues (20)
- Task 'event_events' in the dag 'copy_deduplicate' failing due to table/view not found exception (telemetry.event)) HOT 1
- Tests failing in CircleCI with `ModuleNotFoundError: No module named 'plugins.mozetl'`
- Task `crash_report_parquet` job is failing in socorro_import failing HOT 7
- Add back TAAR lite guids ranking job
- Migrate TAAR jobs from S3 to GCS
- DSRE-22 testing creating issue comment HOT 4
- Test issue for DSRE-22 Jira task HOT 1
- Airflow DAG tag validation is missing
- socorro import uses wrong schema HOT 1
- Add dag definition to schedule webcompat-kb job
- Add a dag definition to schedule broken-site-report-ml job
- Remove gke_command in favor of GKEPodOperator
- Remove GKENatPodOperator
- Create a "Wait for build" operator for dockerized components HOT 1
- Jobs run with GKEOperator need `get_logs=False`, otherwise job is likely to fail unless constantly logging to standard out HOT 2
- glam_org_mozilla_fenix dag failing on gke_command
- prio-processor dag fails to start gke cluster
- Specify exact container version for pipelines feeding public data HOT 1
- Add top-level option to bigquery_etl_query for replacing whole table
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from telemetry-airflow.