AI Decision Making: When AI Should Judge and When It Should Write

AI decision making works best with a model that only returns numbers and labels. When to use a judgment model, when to use a chat model, and the trap between them.

AI decision making means asking a model to pick an answer from a fixed set: a label, a score, a yes or no. Asking AI to write is a different job. Decisions now have their own kind of model.

This guide covers when to use a judgment model and when to use a chat model. It also covers the one trap between them, and a five-minute check for your own prompts.

The short version

If the answer is a number or a label, send it to a judgment model.
If the answer is prose, send it to a chat model.
Counting stays in ordinary code, because neither model counts reliably.
A judgment model cannot refuse. Block requests for writing before they reach it.

Two shapes of answer

Every request you send to AI wants one of two shapes back.

Some want a decision. Pick one option from a fixed list. Say how likely something is to be true. Place it on a scale. Rank a list by the same score.

Others want prose. Anything a person will read. A summary, an explanation, a rewrite, a plan across several steps. Anything in your own voice.

A number, or a paragraph? Ask a judgment model to pick one option from a fixed list, say how likely something is to be true, place it on a scale, or rank items by the same score. Ask a chat model for anything a person will read, to summarise, explain or rewrite, to plan across several steps, or for anything in your own voice.

What a judgment model did in one week

On September 15 a company called TypeSafe released a model called Jev. They call it a System One model, after the fast, instinctive thinking in Kahneman's book. Their docs are plain about the limit: "System One models do not write replies, produce code, or generate explanations of their reasoning."

I gave it three transcripts from my own training sessions. It answered 256 questions about them. Who asked real questions. Who went quiet. What I promised out loud while teaching and then forgot about.

All three transcripts together cost under a cent. Each answer came back in less than half a second.

Then I ran it across 162 paragraphs of my recent drafts. I was hunting a writing habit I keep trying to kill. My plain text search for the obvious phrasing found nothing at all. The model found 27. The top hit, at 98%, was a line from my own newsletter.

It failed at plenty, too. I asked it to guess which newsletter subject lines would land, across ten issues. There was no signal at all, so I dropped that idea the same day.

Their docs also carry a page listing nine things the model is bad at. One line reads "Jev is not a calculator". That page is worth reading before you build anything.

The trap: a judgment model cannot say no

Ask a judgment model to summarise something and it still answers. The answer comes back in the shape you gave it, tidy and meaningless. Nothing errors, so nothing downstream can tell.

So the routing rule needs a guard. I turned mine into a check in my own code. It blocks a writing request before it reaches the judgment model. If your team builds on these models, put the same check in first.

AI decision making examples

These are the kinds of questions that fit a judgment model, all taken from the week above:

Did this participant ask a real question in the session? Yes or no.
Did the trainer promise something out loud? Yes or no.
Does this paragraph set an idea up just to knock it down? A probability.
Which of these items ranks highest on one score? A ranked list.

And one that did not fit: predicting which newsletter subject line will get opened. Across ten issues the model showed no signal at all.

Check one prompt against its answer

Five minutes, in any AI tool you already pay for.

Find one prompt where you ask AI to review, score, rate or sort something. Read what you asked for. Then read what came back.

Say you wanted a number or a label, and it sent three paragraphs. You are paying for writing nobody reads. Ask for the answer on its own. Then put that answer into a rule you can act on.

If you wanted the thinking, keep the paragraphs. That one is the right job for a chat model.

Check one prompt against its answer, in four steps: find one prompt where you ask AI to review, score, rate or sort something; compare what you asked for with what came back; if you wanted a label, ask for the label on its own; put the answer in a rule. If you wanted the thinking, keep the paragraphs.

AI decision making FAQs

What is AI decision making?

It is using a model to choose an answer from a fixed set of options: a label, a score, a probability or a yes or no. The output feeds a rule or a workflow, and no person has to read a paragraph first.

What is a judgment model?

A model that only returns numbers and labels. TypeSafe calls these System One models. They cannot write replies, produce code or explain their reasoning. In my test each answer came back in under half a second.

Should AI do the counting?

No. Counting stays in ordinary code. Neither judgment models nor chat models count reliably, and a script counts the same way every time.

When should I use a chat model instead?

Whenever a person will read the answer. Summaries, explanations, rewrites, plans across several steps and anything in your own voice all belong to a chat model.

Where to start

Run the five-minute check on one prompt this week. For a second look across your team, our free AI Audit shows where your AI use produces results.

If you are ready to wire decisions into real workflows, that routing rule is where our AI implementation consulting starts. We map which questions go to which model, add the guard, and hand the system over to your team.

For the same idea applied to scheduled agents, read AI agent monitoring: why a schedule is not a job. This guide started as issue #234 of The AI Operator Weekly. The newsletter gets the stories a week before they become guides.