AI Agent Monitoring: Why a Schedule Is Not a Job

On a Saturday in September I turned our whole agent fleet back on. Then I read the logs. About 11,000 scheduled runs a week. Zero work done.

Nothing was broken. Every job fired on time and returned a clean result. That is the failure this guide is about. It is the one most AI agent monitoring misses.

This is a short guide to monitoring scheduled AI agents. What to count, what to change, and how often to look.

The short version

AI agent monitoring is the practice of checking that each agent produces outputs a person actually uses. At a cost that makes sense. It is not the same as checking that the agent ran. A run is an event. Work is a result someone reads.

Scheduled agents fail in two directions. A dead agent reports healthy while doing nothing. A busy agent reports activity while producing nothing. Both dashboards look fine. Both are wrong. Monitoring has to catch both. So it compares runs against used outputs. A heartbeat only tells you the agent fired.

What 11,000 runs a week looks like

Here is the real example. On August 14 we paused every scheduled agent job we had. Twenty-five schedules went down to one. On September 12 we turned them back on.

One recovery job was set to run every minute. That is 10,080 runs a week. Every single run returned "disabled" in a few milliseconds. The feature it protects was switched off in production. 1,440 empty calls a day.

Two more jobs polled Slack every ten minutes, looking for answers that were not there. About 500 empty heartbeat rows in three and a half days. Call it 1,000 runs a week.

Add it up. About 11,000 runs a week. Zero outputs a person read.

Two weeks earlier we had the opposite failure. A watchdog reported healthy for 20 days while it was dead. The same dashboard showed both. That is why counting runs tells you nothing on its own.

A schedule is not a job

A job has three parts. A trigger, a result, and someone who reads the result. A schedule only has a clock.

Give an agent a schedule and nothing else. You get one of the two failures above. It runs and does nothing, or it stops and nobody notices. The fix is the same in both cases. Give the job a trigger that means there is work to do. Give it a reader who will miss the result.

Good triggers are events. A button someone clicks. A new file in a folder. A reply in a thread. A form submission. A schedule is fine as a safety net behind a trigger. It is a bad primary.

Two fixes that cut the runs by about 80 percent

The recovery job now runs every five minutes instead of every minute. 288 calls a day instead of 1,440. It still catches everything it needs to catch. The application wakes that worker on demand anyway. The cron was only ever a backstop.

The two pollers stopped polling. Drafting now starts when someone clicks the Done button in Slack. A single daily pass at noon catches anyone who typed "done" instead of clicking. About 1,000 runs a week became about 35.

Nothing in the output changed. The same drafts land in the same place. The only difference is that the log now shows work, and only work. The whole change was a schedule string and a button handler.

The monitoring cadence that replaced a review

We had planned a one-off review of nine trial agents. Sign or refuse, seat by seat. It never happened, because a one-off review is itself a schedule with no trigger. What replaced it is a cadence with an owner.

Mia, our chief-of-staff agent, owns the roster. Four rituals sit around it:

• Daily: clear the approvals digest. Queue depth is the health signal. A growing queue means agents are producing more than a person can read.
• Weekly: a fleet review with a per-seat table of runs, outputs used, and cost.
• Monthly: one seat checked against its own operating brief. Does what it produced match what it was asked to produce?
• Quarterly: cost against contribution for every seat. Anything that costs more than it contributes goes back to a trigger or gets retired.

This is the same shape as managing people. We wrote it up before, in how we manage AI agents with job descriptions and KPIs. Monitoring is the part of that system that runs every day.

Count the runs: a ten-minute check

Pick one scheduled AI job. Open its log. Then:

1. Count the runs in the last seven days. Every fire of the clock, whatever it returned.
2. Count the outputs a person actually read or used. A draft opened. A digest answered. A file someone touched.
3. If the second number is zero, give the job a trigger instead of a clock. A button. A new file. A reply. Something that means there is work to do.

Do this for every scheduled job once, then fold it into the weekly review. The ratio of runs to used outputs is the one AI agent monitoring number to watch. It catches both the dead agent and the busy one.

Frequently asked questions

What is AI agent monitoring?

AI agent monitoring is checking that each AI agent produces outputs a person uses. At an acceptable cost and error rate. It compares runs against used results. A heartbeat only tells you the agent fired.

Should AI agents run on a schedule or a trigger?

A trigger, wherever one exists. An event such as a new file, a reply, or a button click means there is work. A schedule is a good safety net behind a trigger. It is a poor primary, because it runs whether or not there is work.

What should an AI agent monitoring dashboard show?

Per agent: runs in the period, outputs a person read or used, and cost. Plus the last time a human touched the output. The ratio of runs to used outputs is the headline number. A high run count with a zero in the outputs column is the busy-but-idle failure.

How often should you review AI agents?

Daily for the approvals queue. Weekly for the per-agent run and cost table. Monthly for one agent checked against its brief. Quarterly for cost against contribution. A one-off review tends not to happen. A cadence with an owner does.

Where to start

If you have never counted runs against used outputs, do that first this week. For a second look at the whole fleet, our free AI Audit shows where your AI use produces results.

If the fleet is already bigger than one person can watch, that count is where our AI implementation consulting starts. We write the operating brief, the triggers and the review cadence with your team, then hand it over.

This guide started as issue #233 of The AI Operator Weekly. The newsletter gets the stories a week before they become guides.