When NOT to use an AI agent: a checklist for ops leaders

Half of the demos we run end with the same answer: "this is not an agent workflow yet." That is a feature. Spending three months on the wrong workflow is the most expensive mistake an ops leader can make in this space, and saying no early is the cheapest service we provide.

Here is the six-question checklist we use to filter "agent-ready" from "not yet."

1. Is the volume above 500 cases per week?

Below 500 a week, the eval harness will not converge inside a 30-day pilot. You will not have enough labels to validate per-segment accuracy, and the curve will look noisy. Wait until volume is there, or start with a sister workflow that has it.

2. Are the inputs structured?

If 80% of the input data is in defined fields (customer record, ticket fields, claim header), yes. If 80% is free-form (a 4-paragraph email, a hand-written form, a doctor's note), the agent can still work, but you need a document understanding layer, and the timeline doubles.

3. Are the decisions enumerable?

How many distinct decision types are there? If under 25, an agent ships in 30 days. If 25–100, plan 60. If over 100, the workflow is not "automate it with an agent", it is "redesign the process first."

4. Is the action reversible?

Refunds reverse. Address changes reverse. Tickets reopen. Vendor records roll back. If the action is reversible end-to-end, the agent can take it autonomously above threshold. If the action is irreversible (sending a public communication, executing a trade, prescribing a drug), every decision goes through a human, every time.

5. Is there a single ops leader on the hook?

This is the question that kills more pilots than any technical issue. If "the project" is owned by a steering committee, by IT, or by an executive sponsor with no day-to-day involvement, the rollout will stall in coordination. Pick a workflow with a named ops leader who has authority over thresholds, exceptions, and escalations.

6. Does the team review at least 200 cases a week today?

If the answer is no, you do not have enough supervisor capacity to feed the eval harness. The agent's accuracy will plateau at the model's pretrained ceiling. Start with a workflow where reviewers are already in the loop and expand outward.

What to do if you fail one or more

You probably do not need to walk away from AI in your ops stack. You need to start somewhere else. Most teams that fail this checklist on their first-pick workflow have a tier-1 ticket triage queue, a vendor onboarding process, or a claims review backlog that would pass, they just had not framed it as the right candidate.

The right answer is rarely "no AI." The right answer is "AI on a different workflow first." Start where you score 24+ on the rubric. Earn the political capital. Then come back to the harder one.