Vivere

Celery beat

Beat queues the task and a worker runs it. Either half can stop without the other noticing.

A periodic Celery task needs two processes: beat decides it is time and puts a message on the queue, and a worker takes it off and runs it. Either can stop without the other noticing. Beat dying means the queue simply stays empty, which looks exactly like a quiet night. A worker dying means the queue grows and nothing runs, and neither produces the task failure your task_failure handler or Sentry integration is watching for.

Ping from the task, not from beat

A ping sent by beat would only prove the timer fired. A ping sent by the task proves beat scheduled it and a worker executed it, which is the thing you actually care about.

import urllib.request
from functools import wraps

PING = "https://vivere.dev/p/<monitor-id>"

def ping(suffix="", body=None):
    try:
        urllib.request.urlopen(PING + suffix, data=body, timeout=10)
    except Exception:
        pass

def monitored(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        ping("/start")
        try:
            result = func(*args, **kwargs)
        except Exception as exc:
            ping("/fail", str(exc).encode())
            raise
        ping("", str(result).encode() if result else None)
        return result
    return wrapper

@app.task
@monitored
def nightly_report():
    ...

Order matters: @app.task on the outside, so Celery registers the wrapped function.

Beat schedules

from celery.schedules import crontab

app.conf.beat_schedule = {
    "nightly-report": {
        "task": "tasks.nightly_report",
        "schedule": crontab(hour=2, minute=0),
    },
}

With django-celery-beat the schedule lives in the database instead, where a PeriodicTask row can be switched to enabled=False by anyone with admin access and stay that way for months. That is the case a heartbeat catches and a code review does not.

Monitor settings

Use the same cron expression as the beat entry, with CELERY_TIMEZONE. If the task has autoretry_for or retry_backoff, a run that eventually succeeds can ping well after its schedule, so set the grace from the worst retried run rather than the happy path. The /fail ping fires on the first failure, before the retries; if you would rather only hear when the retries are exhausted, move it into an on_failure handler on a bind=True task and check self.request.retries.