run_failure_sensor and run_status_sensor for DagsterRunStatus.FAILURE not working on 1.7.3 about dagster

[gibsondan]

@marcilj I tried loading this file based on your example:

from dagster import (
    DefaultSensorStatus,
    Definitions,
    RunFailureSensorContext,
    RunRequest,
    job,
    op,
    run_failure_sensor,
)


@op
def fails():
    raise Exception("failure!")


@job
def my_job_fails():
    fails()


@run_failure_sensor(
    request_job=my_job_fails, minimum_interval_seconds=5, default_status=DefaultSensorStatus.RUNNING
)
def my_failure_sensor(context: RunFailureSensorContext):
    return RunRequest(job_name="my_job_fails")


defs = Definitions(sensors=[my_failure_sensor], jobs=[my_job_fails])

• ran dagster dev
• Launched the job, saw it fail, and viewed the sensor timeline

I can see the run being launched as expected in 1.7.3:

It sounds like that is different than what you are seeing - if you run those same steps do you get different results? Any other details or differences that might be helpful for us to reproduce what you're seeing?

[marcilj]

Oh damm... you are totally right.

Tried it with that basic code, and ended up with a working sensor.

Ok, so found the issue, and this might be a feature. I was testing my sensor on a failed job from another Code Location.
What throw me off was that the record_timestamp was getting refreshed, but nothing else was happening.

Can you please tell me why would the record_timestamp get updated if the sensors technically shouldn't look at failed jobs from another code location?

Adding monitor_all_code_locations=True in @run_failure_sensor( request_job=send_event_to_datadog_job, monitor_all_code_locations=True ) fixed the issue.

Thank for the quick reply.

[gibsondan]

No problem - record_timestamp gets updated because the sensor is tailing the full event stream and filtering out invalid events.