Vivere

Vivere / Answers

Kubernetes CronJob missed its schedule

A CronJob that misses 100 schedules stops scheduling entirely and says so only in an event you have probably already lost.

A CronJob that stops creating Jobs usually explains itself in a Kubernetes event — which is garbage collected about an hour later, long before anybody looks. Here is what to check and what the controller is actually doing.

1. Is it suspended?

kubectl get cronjob nightly-export -o jsonpath='{.spec.suspend}{"\n"}'

true means the controller will not create Jobs. CronJobs get suspended by deployment tooling, by a rollback to an older manifest, and by hand during an incident that nobody remembered to undo.

2. Has it missed more than 100 schedules?

This is the failure mode that surprises people. On each pass, the controller counts how many start times it missed since status.lastScheduleTime. If that count exceeds 100, it stops scheduling entirely and records:

Cannot determine if job needs to be started:
too many missed start times (> 100). Set or decrease .spec.startingDeadlineSeconds
or check clock skew

A CronJob running every minute reaches 100 misses in under two hours of control-plane trouble; one running every five minutes reaches it in a working day. Once it is in this state it never recovers on its own.

kubectl get events --field-selector involvedObject.name=nightly-export
kubectl get cronjob nightly-export -o jsonpath='{.status.lastScheduleTime}{"\n"}'

Setting spec.startingDeadlineSeconds bounds how far back the controller counts, which both prevents the runaway count and stops it firing a backlog at you. A value somewhat shorter than the interval is usual. Recreating the CronJob clears a stuck one.

3. Is the previous run still going?

With concurrencyPolicy: Forbid, a schedule that arrives while the previous Job is still running is skipped, not queued. A job that slowly gets slower will eventually overrun its own interval and then appear to run at half the rate, then a quarter, with no error anywhere. Replace kills the old one instead; Allow, the default, lets them overlap.

Pair this with activeDeadlineSeconds on the Job spec so a hung run cannot block every subsequent one indefinitely.

4. Which timezone is the schedule in?

Without spec.timeZone, the schedule is interpreted in the timezone of the kube-controller-manager, which in practice is UTC. Since Kubernetes 1.27 you can set it explicitly:

spec:
  schedule: "0 2 * * *"
  timeZone: "Europe/Oslo"

5. The Job was created but the pod never ran

If kubectl get jobs shows the Job but nothing completed, the CronJob did its part and the failure is below it: insufficient CPU or memory on every node, a node selector or taint that nothing satisfies, ImagePullBackOff on a tag that was deleted, a missing Secret or ConfigMap, or a PVC that cannot bind.

kubectl describe job nightly-export-29414880 | tail -20
kubectl get pods --selector job-name=nightly-export-29414880

6. The evidence has been deleted

successfulJobsHistoryLimit defaults to 3 and failedJobsHistoryLimit to 1. On a frequent schedule, the Job that failed at 02:00 is long gone by morning, along with its pod and its logs. Raise the failed limit on anything you expect to debug, and ship the logs somewhere outside the cluster.

Alerting on the run, not on the pod

Everything above is invisible to a monitor that watches pods, because in most of these cases there is no pod to watch. What you want to know is "did the work happen", and only the work can answer that.

Have the container report success on its way out:

spec:
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: export
              image: registry.example.com/export:1.4.2
              command: ["/bin/sh", "-c"]
              args:
                - /app/export.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://vivere.dev/p/$(MONITOR_ID)
              env:
                - name: MONITOR_ID
                  valueFrom:
                    secretKeyRef: { name: vivere, key: monitor-id }

Set the monitor's period to the schedule and its grace to the longest acceptable delay. Suspension, the 100-miss lockout, a skipped run under Forbid, an unschedulable pod and a failing image all produce the same symptom — no ping — and all raise the same alert.

Common questions

Why did my Kubernetes CronJob not create a Job?

The usual causes are that the CronJob is suspended, that concurrencyPolicy is Forbid and the previous Job is still running, that startingDeadlineSeconds elapsed while the controller was unavailable, or that the CronJob has missed more than 100 schedules and the controller has given up on it.

What happens after 100 missed schedules?

The CronJob controller stops scheduling and records a warning event saying it cannot determine when it last ran. Because events are garbage collected after about an hour by default, the explanation is usually gone by the time anybody looks. Deleting and recreating the CronJob resets it.

What timezone does a CronJob use?

The kube-controller-manager's timezone, which is normally UTC, unless the CronJob sets spec.timeZone (available since Kubernetes 1.27). A schedule written in local time without that field runs at a different hour than intended.

Find out without looking

Vivere watches for the ping that does not arrive. Add one line to the job, pick where alerts should land, and you hear about the run that never happened.

Start free Read the quickstart

Ten monitors, a status page, email and webhook alerts. No card.

Last reviewed September 2026.

Related