A convincing AI demonstration is not proof that the task belongs in production. Three clean examples can hide missing information, ambiguous decisions and expensive mistakes. Before choosing a model, decide what useful work the feature would actually complete.
Selection is cheap to get right. Four questions do most of the work.
Is the volume high enough to matter
A hosted model can incur usage charges, while the integration needs implementation and continuing maintenance. Low volume alone does not settle the decision: an infrequent task can still be costly or difficult. Count its frequency, the work required and the consequences of delay before estimating whether automation is worthwhile.
The tasks worth automating are the ones nobody mentions because they are too routine to raise. Sorting inbound messages. Pulling four fields out of a document. Summarising a call. Drafting the same reply with different details. Each takes a few minutes and each happens hundreds of times.
Start by counting rather than by asking. Timestamps in your systems will tell you where the volume is, and it is frequently not where people say it is.
Can the output be checked
This is the question that eliminates most candidates.
If a person can look at the output and know within seconds whether it is right, you have a workable task. Classification is checkable. Extraction is checkable against the source document. A draft reply is checkable by the person about to send it.
If checking the output requires as much work as producing it, the model has saved nothing. If nobody can tell whether it is right at all, you have built something that generates confident text of unknown value, which is worse than nothing because people will act on it.
The corollary is that human review should be designed in at the point where the answer stops being checkable, not bolted on afterwards.
Is the data already reachable
A model is only as good as what it can see. Before anything is built, establish where the information lives, whether it is current, whether it is structured enough to retrieve against, and whether the people who will use the feature are permitted to see all of it.
Three common failures live here. The documents exist but are scanned images with no extracted text. The records exist but in a system with no interface we can read. The data exists and is current but nobody has ever decided who is allowed to see which parts, which means permission filtered retrieval has no rules to apply.
Each of those is fixable. Each of them is also a project of its own, and discovering it after committing to a delivery date is how a pilot becomes an embarrassment.
What happens when it is wrong
Every candidate needs an answer to this, and the answer determines the design.
If a wrong classification routes a message to the wrong queue and a person notices in an hour, the cost is low and full automation is reasonable. If a wrong extraction writes an incorrect figure into an invoice, the cost is high and the design needs a validation step, a confidence threshold and a review queue.
If the honest answer is that nobody would notice a wrong answer, that is not a reason to proceed. It usually means the output is not being used, which means the task did not need doing.
When the answer is ordinary code
A significant proportion of work described as an AI problem is a rule.
If the routing depends on which form was submitted, that is a rule. If eligibility depends on three fields and a threshold, that is a rule. If the summary is always the same five values in the same order, that is a template.
Rules are cheaper, faster, testable, explainable and they do not change their behaviour when a provider updates a model. Use a model where the distinction genuinely requires judgement about unstructured input, and use code everywhere else. A system that uses a model for something a rule could do is paying for latency and uncertainty it did not need.
Scoring, not arguing
Put every candidate in a table with those four answers plus the estimated volume and the cost of a wrong answer. The shortlist emerges from the table rather than from the meeting, and the rejected candidates have a written reason, which stops them being reproposed every quarter.
Then build an evaluation set from real cases before writing the feature. A small initial set can expose obvious problems, but it is not statistical proof of reliability. Include the unusual cases as well as the routine ones, and expand the set as new failures appear.
Put a candidate through one complete shift
Consider an assistant that sorts requests arriving in a contractor's shared inbox. Define the allowed categories first: new service request, existing appointment question, supplier message and manual review. Do not require the model to guess a category when a message contains two unrelated requests. That ambiguity needs an explicit route.
Run the candidate alongside the current process without letting it move messages. Compare its proposed routing with the dispatcher's actual decision, and ask about disagreements. A different answer may reveal an unclear business rule rather than a model failure. For example, a request mentioning a previous visit might be a new job or a question about an existing one. The relevant distinction is whether dispatch can identify the appointment, not whether the email contains the word visit.
Record review time as well as correct classifications. If dispatchers must open attachments, search another system and rewrite the answer, the apparent automation may only have moved their work. Include unanswered messages and review backlog in the comparison so that abstaining on everything cannot look like success.
The first release can remain a recommendation beside the inbox. Give dispatchers a way to correct the category and record a reason without writing a separate report. Those corrections become useful evaluation examples. Expand authority only after the team understands which cases the feature handles and which still require a person.