How I Manage 120+ AI Agents (Without Losing Control)

My first AI agent was a disaster.

I built it to handle outreach emails. Gave it a prompt, connected it to my CRM, and let it run. Two days later, a prospect forwarded me the email my agent had sent them. It had hallucinated a case study that didn't exist, and quoted pricing I'd never offered.

That was the moment I realized something. AI agents aren't magic. They're workers who need management.

The Problem with "Deploy and Hope"

Most people treat AI agents like vending machines. Put in a prompt, get out a result. If the result is bad, tweak the prompt and try again.

This doesn't scale. When you have one agent, you can babysit it. When you have ten, you can't. When you have 120 (which is where I am today), you need a system.

The system I built borrows from something every manager already knows: people management. Every agent I run has four things: A job description. KPIs. A QA process. And a firing threshold.

How to Write an Agent Job Description

In my system, every agent has a job description in its custom instructions that defines exactly what the agent does, what tools it can access, what rules it follows, and what it should never do.

Agents also have memory, a set of files it can create and update as it works, and access to the “brain” — a shared knowledge base that every agent in the system can query, keeping outputs consistent no matter which agent produced them.

A good agent job description covers five things:

1. Scope — What is this agent responsible for? Be specific. "Draft the weekly newsletter using this week's research brief, following brand voice guidelines, under 800 words" is a good start to a job description.
2. Tools and access — Which systems can this agent touch? A newsletter agent needs access to the research brief folder and the drafts folder. It does not need access to the CRM or the financial data. Limit scope to limit risk.
3. Rules and constraints — What must the agent always do? What must it never do? My writing agents have a list of forbidden phrases. My data agents have rules about never fabricating numbers. Write these down explicitly. Agents follow instructions literally. If you don't write the rule, the rule doesn't exist.
4. Output format — What should the deliverable look like? Define the structure, length, format, and naming conventions. If the output doesn't match the spec, you know immediately something went wrong.
5. Escalation triggers — When should the agent stop and ask for human input? Define the edge cases. "If the data source is unavailable, stop and flag. Do not generate placeholder data."

The more precise the job description, the more consistent the output. I've found that most agent failures trace back to vague instructions, not bad models.

Setting KPIs for AI Agents

Every agent in my system gets measured on four metrics:

• Completion rate — Did the agent finish the task? This sounds basic, but agents fail silently more often than you'd expect. They time out. They hit an error and produce partial output. They get stuck in a loop. Track whether the job actually got done.
• Accuracy — Is the output correct? For a research agent, did it pull the right data? For a writing agent, did it follow the style guide? For a data agent, do the numbers match the source? Accuracy requires spot checks, which I'll cover in the QA section.
• Speed — How long did the task take? Not because faster is always better. Because sudden changes in speed signal problems. If an agent that normally completes in 30 seconds starts taking 5 minutes, something changed. The data source might be different. The prompt might be hitting edge cases. Speed is a diagnostic signal.
• Cost per task — Every API call costs money. Track what each agent costs to run per task. This matters less for occasional tasks and more for high-volume agents. My lead enrichment agents run hundreds of times a day. A 10% increase in tokens per run adds up fast.

You don't need a fancy dashboard for this. A simple spreadsheet works. Log the metrics weekly. Look for trends. Flat is good. Spikes need investigation.

How to QA Your Agent Output

KPIs tell you what happened. QA tells you whether it was good enough.

I run three types of QA on my agents.

Spot checks are the foundation. Pick a random sample of outputs each week and review them manually. For high-volume agents, I check 5-10% of outputs. For critical agents — anything client-facing — I check every single output before it ships. There's no shortcut here. You have to read the work.

Rule compliance is where you automate. Did the output follow the rules defined in the job description? I run compliance checks that scan for forbidden phrases, verify format requirements, and flag outputs that break constraints. Think of this as the automated layer of QA.

Outcome tracking is the final test. Did the output achieve its goal? A newsletter draft that passes all format checks but gets zero engagement still failed. Track downstream outcomes where possible — open rates, response rates, accuracy rates. The output can be technically correct and practically useless.

Together, these three layers give you a complete picture. Spot checks catch quality issues. Rule compliance catches process issues. Outcome tracking catches relevance issues.

Not sure how well your current AI setup is performing? An AI Audit will show you exactly where the gaps are.

When to Fire an Agent

Sometimes an agent isn't working, and no amount of prompt tweaking will fix it.

I fire an agent when any of these are true:

• Accuracy drops below 80% consistently. If one in five outputs is wrong, the agent is creating more work than it saves. You're spending time finding and fixing errors instead of doing the actual work.
• The QA burden exceeds the time saved. If reviewing the agent's output takes almost as long as doing the task yourself, the agent isn't adding value. It's just adding a step.
• The task has changed. The agent was built for a specific job. If the job evolved, the agent might not fit anymore. Don't patch an old agent endlessly. Retire it and build a new one for the new job.
• The model improved. AI models get better over time. An agent I built six months ago might be running on outdated patterns. Sometimes the best move is to start fresh with the latest model and new instructions, rather than carrying forward old workarounds.

Firing an agent isn't failure. It's iteration. I've retired hundreds of agents over the past year. Each one taught me something about how to build the next one better.

Treat Your Agents Like a Workforce

The biggest shift isn't technical. It's mental.

Most people think about AI as technology. I think about it as workforce. Technology you configure once and forget. A workforce you manage continuously.

That means documentation. Every agent has a file. Every change gets logged. When something breaks, I can trace it back to what changed and why. Without that, you're debugging in the dark.

Running 120+ agents sounds complex. And it is. But it's manageable, because every agent follows the same system, and the system doesn't change.

Try This Today: One Agent, One Job Description

If you're running even one AI agent today, try this.

Write a one-page job description for it. Define exactly what it does, what it can access, and what it should never do. Set three measurable KPIs. Run a spot check on its last ten outputs.

You'll probably find gaps. Instructions that are too vague. Outputs that drift from what you intended. Metrics you've never tracked.

That's normal. That's where management starts.

The agent that embarrassed me in year one is long gone. What replaced it is a workforce I trust, because I manage it. Clear job descriptions. Measurable KPIs. Regular QA. A willingness to make the hard call when something isn't working. That's the whole system.

If you're building out your AI workforce and want to get the structure right from the start, book a strategy call — we'll help you build something you can actually rely on.