Vivere

Vivere / Answers

Catching up a missed cron job without getting the dates wrong

Most schedulers skip a run that was due while they were down, and the ones that catch up run it once, late. A job that works out its dates from the clock then processes the wrong span.

A scheduled job that did not run has two problems. The first is finding out. The second arrives once you know: the host or the scheduler is back, and something has to decide whether to run the job late and, if it does, which data the late run should cover. Get that wrong and the catch-up run does damage that nobody traces back to the night it happened.

What each scheduler does with a missed run

Few schedulers catch up at all, and none of them runs a job more than once to make up for several missed runs.

So the realistic outcomes are a run that never happens, or a single run at an unexpected time standing in for one or more that were missed. A job is only safe under a catch-up setting if it does the right thing in both.

Why a late run covers the wrong span

Most scheduled jobs work out what to process from the clock at the moment they start. A nightly export selects rows from the last 24 hours. A report names its file with today's date. A summary covers "today" or "yesterday". Each of those is right only when the job runs at the time it was written for.

Take an export that runs at 02:00 and selects the last 24 hours. The run due at 02:00 on the 16th is missed while the host is being patched, and a catch-up run starts at 11:00. It exports from 11:00 on the 15th to 11:00 on the 16th. The last successful export ended at 02:00 on the 15th, so the rows from 02:00 to 11:00 on the 15th are in no export at all. The rows from 02:00 to 11:00 on the 16th will be exported again by the next run. Nothing errors, and the totals are wrong by nine hours of data in both directions.

Names taken from the clock fail the same way across midnight. A job that runs at 23:50 to report the day's numbers, caught up at 00:20 after a restart, reports the new day with twenty minutes of data in it, and the day that just ended is never reported. If it writes report-$(date +%F).csv, that file carries the new day's name, and the on-time run that night overwrites it.

Key the work by window, not by the time it runs

The fix is to take the span from the schedule rather than from the clock. A daily job covers whole days, and the days a run should process are every day that has ended and has not been done yet. When the run starts stops mattering.

That needs one piece of state: which windows are done. A small table is enough. This is PostgreSQL, but any database with a unique key will do:

CREATE TABLE job_windows (
  job          text        NOT NULL,
  window_start timestamptz NOT NULL,
  claimed_at   timestamptz NOT NULL DEFAULT now(),
  finished_at  timestamptz,
  PRIMARY KEY (job, window_start)
);

The job then works through the windows it owes, oldest first:

from datetime import datetime, timedelta, timezone

DAY = timedelta(days=1)

def windows_due(last_done, now):
    """Every whole UTC day after last_done that has ended by now."""
    today = now.astimezone(timezone.utc).replace(hour=0, minute=0, second=0, microsecond=0)
    start = last_done + DAY if last_done else today - DAY
    while start + DAY <= today:
        yield start, start + DAY
        start += DAY

def run(db):
    last_done = db.last_finished_window("daily-report")  # None on the first run
    for start, end in windows_due(last_done, datetime.now(timezone.utc)):
        if not db.claim("daily-report", start):  # another run holds it
            break
        build_report(start, end)  # created_at >= start AND created_at < end
        db.finish("daily-report", start)

last_finished_window is SELECT max(window_start) ... WHERE finished_at IS NOT NULL, and finish sets finished_at. A run at 02:00 does yesterday. A run at 11:00 after a missed night does the same day with the same boundaries. A run after the host was down for three days does all three, one at a time, where a scheduler's catch-up setting would have run the job once.

Make doing a window twice harmless

The table makes a repeat detectable, but a crash between finishing the work and recording it still repeats that window on the next run. The work for one window has to survive being done twice:

The same table stops two triggers doing one window at the same time, which is what makes a second trigger on another machine safe to add. claim inserts the window and works on it only if a row comes back:

INSERT INTO job_windows (job, window_start)
VALUES ('daily-report', $1)
ON CONFLICT (job, window_start) DO UPDATE
  SET claimed_at = now()
  WHERE job_windows.finished_at IS NULL
    AND job_windows.claimed_at < now() - interval '2 hours'
RETURNING window_start;

No row back means another run holds the window or has already finished it. The WHERE clause lets a run take over a claim older than the job's longest normal run, so a run that crashed halfway does not block its window forever. Set that interval from real durations with room to spare, or two slow runs will both do the window.

A gap now shows in your own data

Once every window leaves a row, a missed one can be found with a query. This lists the days in the last two weeks with no finished run:

SELECT d::date AS missing_day
FROM generate_series(current_date - 14, current_date - 1, interval '1 day') AS d
WHERE NOT EXISTS (
  SELECT 1 FROM job_windows w
  WHERE w.job = 'daily-report'
    AND w.finished_at IS NOT NULL
    AND (w.window_start AT TIME ZONE 'UTC')::date = d::date
);

That is worth having. It still needs something to run it on a schedule and to tell someone when it returns a row, and if that is a cron entry on the same host or a workflow on the same n8n instance, it stops when the job does. The gap it was written to find then goes unreported for the same reason the job did not run.

Hearing about the gap from outside

The check that windows are being done has to live somewhere the job's failures cannot reach. Have the job call a heartbeat URL each time it finishes a window, and let a missing call be the alert:

import urllib.request

PING = "https://vivere.dev/p/<your-monitor-id>"

# after db.finish("daily-report", start)
urllib.request.urlopen(PING, data=f"finished {start:%Y-%m-%d}".encode(), timeout=10)

If the host is down, the scheduler skipped the run, or the job is stuck behind a claim, the call does not arrive and the monitor alerts. The alert is also your prompt to check, once things are back, that the catch-up covered every window. The body of each call is kept as the run's log, so the monitor's history shows which day each run finished. Set the monitor's schedule to the job's and its grace from the job's slowest normal run; the grace period calculator does that arithmetic. Vivere's free plan covers ten monitors.

Common questions

Does cron run missed jobs after a reboot?

No. Cron checks every minute whether an entry matches the current time, and a minute that passed while the machine was off or cron was stopped is never looked at again. anacron catches up daily, weekly and monthly jobs on machines that are not always on, and a systemd timer with Persistent=true starts a missed service once when the timer is active again.

What does idempotent mean for a scheduled job?

Running it twice for the same window leaves the same result as running it once: no duplicate rows, no second email, no file for one day overwritten with another day's data. It is what makes a catch-up run, a manual retry and two overlapping triggers safe.

Should a missed run be caught up or skipped?

It depends on whether the job is about a period or about the present. A daily report, a billing export or a data sync covers a window, and skipping it leaves a hole, so catch it up for the window it missed. A cache refresh or a job that publishes the latest value is about now, and running it for an old window achieves nothing, so let the next scheduled run do it.

Find out without looking

Vivere watches for the ping that does not arrive. Add one line to the job, pick where alerts should land, and you hear about the run that never happened.

Start free Read the quickstart

Ten monitors, a status page, email and webhook alerts. No card.

Last reviewed September 2026.

Related