Kubernetes CronJob Monitoring

What this guide covers

Kubernetes will tell you whether a CronJob was scheduled and whether its pods exited zero — but not whether the job actually did its job. This guide covers monitoring the business outcome of a CronJob with an external heartbeat, why cluster status alone is not enough, and how to pair the two for fast debugging when something is late.

• Object status shows scheduling, not business success.
• An external heartbeat survives a cluster-wide problem.
• Pair the heartbeat with cluster events for debugging.

Why in-cluster status is not enough

The CronJob controller records successful and failed job runs, and you can watch pod phases and exit codes. But this tells you about the mechanics, not the meaning: a job can be marked successful while doing nothing useful, and — more dangerously — if the controller itself, the nodes, or the cluster network is down, there is nothing to emit an alert at all. Monitoring that lives inside the thing you are monitoring cannot report on its own failure.

Adding an external heartbeat

Put a single call at the end of your job's happy path — after the export is written, the reconciliation is committed, whatever "done" means for this job. Because Sandglass is external, it holds the source of truth: if the heartbeat is late, something is wrong, whether that is a failed job, a paused CronJob, or a cluster-wide outage. When it alerts, use `kubectl get cronjob`, job history, and events to diagnose which of those it is.

• Call the heartbeat as the container's final successful action.
• Set the interval to the schedule plus a grace buffer for scheduling skew.
• On alert, inspect CronJob status, job history, and pod events to localize the cause.

How Sandglass supports the practice

Have the CronJob's container call a Sandglass heartbeat URL after the work completes successfully, with the interval set to the schedule plus a grace buffer. Because the signal originates outside the cluster, it still fires an alert when the whole cluster or the CronJob controller is unhealthy — the exact situation where in-cluster monitoring goes dark too.

• Back the practices here with HTTP, ping, TCP, content, SSL certificate, and heartbeat checks.
• Route incidents to email, Slack webhook channels, and generic webhooks so the right people respond fast.
• Use a public status page to keep customers informed while the team works the incident.

Common mistakes to avoid

Kubernetes status can show scheduling history and pod exit codes, but a pod exiting zero does not prove the job reached its success path — it may have skipped work, hit an empty queue, or logged an error and exited cleanly anyway. An external heartbeat placed after the real work is what proves the outcome, not just the process.