Comments (2)
Just a note to those who are reading this and wondering what to do about the time zone. This can be set in DailyPartitionsDefinition. @garethbrickman thanks for the detailed explanation.
from dagster.
The reason you cannot specify both the cron_schedule/execution_timezone
parameters and the day_of_month/day_of_week/hour_of_day/minute_of_hour
parameters at the same time when creating a schedule from a partitioned job is due to the design of the Dagster scheduling system. The build_schedule_from_partitioned_job
function is designed to create a schedule that aligns with the partitioning of the job. This means that the schedule's frequency and timing should match the intervals defined by the job's partitions.
When you define a partitioned job, you are effectively setting up a structure for how the data should be processed over time. For example, if you have a job that processes data daily, you would set up daily partitions. The schedule for this job should then naturally run once per day to process each partition.
The day_of_month
, day_of_week
, hour_of_day
, and minute_of_hour
parameters are used to specify the exact time within the partition's interval when the job should run. These parameters are meant to provide a convenient way to set the schedule based on the partition's natural frequency without having to manually calculate the corresponding cron expression.
On the other hand, the cron_schedule parameter
allows you to specify a custom cron expression that defines the schedule's frequency and timing in a more granular and flexible way. The execution_timezone
parameter is used in conjunction with cron_schedule
to ensure that the schedule runs at the correct local time, taking into account time zone differences.
Allowing both sets of parameters to be specified would create ambiguity and potential conflicts in the scheduling logic. For instance, if you set a daily partition but then provide a cron expression that runs every hour, it would be unclear how the schedule should reconcile the hourly cron expression with the daily partitioning of the job.
To maintain clarity and prevent such conflicts, Dagster requires you to use either the partition-based parameters or the cron-based parameters , but not both. This ensures that the schedule is consistent with the partitioning of the job and that there is a single source of truth for when and how often the job should run.
from dagster.
Related Issues (20)
- CeleryK8sRunLauncher doesn't work with celery_executor HOT 3
- Hooks: slack_on_success and slack_on_failure fail silently to work
- [dagster-deltalake] GcsConfig ImportError and TypeError for partitioned assets
- Different UX when viewing runs locally than in production because of additional tags
- `load_asset_checks_from_module` sometimes returns `AssetsDefinition`s instead of `AssetChecksDefinition`s
- ModuleNotFoundError: No module named 'dbt.adapters.base.impl' HOT 3
- Support tags with colon (:) HOT 2
- Propagate filters when navigating through catalog search results
- dagster_pipes.DagsterPipesError: Cannot send message after pipes context is closed." HOT 2
- Cannot create asset job with BackfillPolicy.multi_run() and backfill_policy=None
- GCS IO manager connection error "AttributeError: 'NoneType' object has no attribute 'get_client'" HOT 3
- Dagster-pipes report_asset_materialization is missing `partition` versioning
- Daemon error on code location reload HOT 3
- Dagster-pipes errors out even though run was successfull HOT 1
- Allow easily backfilling all failed partitions for different assets with different partitions HOT 2
- @io_manager(config_schema=...) fails to resolve silently after pydantic update >=2.7.1
- Provide more flexible launch-time configuration composition options for Kubernetes
- Allow multiple partition selection in launch pad
- Launchpad on asset shows all resources instead of the required ones HOT 2
- Missing env var causes silent failure of materialization from UI HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagster.