Dagster

Schedules are evaluated by the daemon. If the daemon is down or the schedule is stopped, there is no run to fail.

Schedules and sensors in Dagster are evaluated by dagster-daemon, a separate process from the web server. If the daemon is not running, the UI still shows the schedule as running and nothing ticks: no runs, no failures, no alerts. A schedule left stopped after a code location reload does the same thing, and so does a code location that fails to load.

Ping from inside the run

Put the ping in the job itself, not in a sensor. A sensor needs the daemon, which is one of the things that can be missing.

import urllib.request
from dagster import In, Nothing, job, op

PING = "https://vivere.dev/p/<monitor-id>"

def ping(suffix="", body=None):
    try:
        urllib.request.urlopen(PING + suffix, data=body, timeout=10)
    except Exception:
        pass

@op(ins={"after": In(Nothing)})
def ping_vivere():
    ping()

@job
def nightly_etl():
    ping_vivere(after=load(transform(extract())))

For assets

from dagster import AssetExecutionContext, asset

@asset(deps=[orders_cleaned])
def orders_reported(context: AssetExecutionContext):
    ping("", f"materialised at {context.run_id}".encode())

Failures

from dagster import DagsterRunStatus, RunFailureSensorContext, run_failure_sensor

@run_failure_sensor(monitored_jobs=[nightly_etl])
def report_failure(context: RunFailureSensorContext):
    ping("/fail", context.failure_event.message.encode())

This one is worth having as a sensor, because it only has to work when runs are happening, and it is the fast path: the monitor goes Down straight away with the message attached instead of waiting out the deadline.

Monitor settings

Use the schedule's own cron string and its execution_timezone. Dagster queues runs when the run coordinator limits concurrency, so the grace should cover the queue wait as well as the job.