A team starts an AI pilot after a promising demo. A few tasks look faster. Then the owner asks the harder question: did the AI pilot improve the work, or did the team simply move effort somewhere else?
An AI pilot can answer that question, but only if the business records a fair starting point first. Without a baseline, teams often compare a polished test with a messy memory of the old process. The result may sound convincing while saying little about service quality, total cost or the work employees still have to do. A baseline gives the AI pilot a fair comparison.
A baseline is a short, agreed picture of how a defined task works today. It gives the team something concrete to compare with the pilot. It does not need a complex analytics project. It needs measures that match the business decision.
Start an AI pilot with one task and one decision
Do not begin with a broad goal such as “use AI across customer service.” Name one recurring task. As an illustrative example, a team may use AI to prepare a first draft of replies to routine delivery questions. The system might suggest language, while an employee checks and sends each response.
Then state the decision the AI pilot must support. Are you deciding whether to expand the task to another team, change the review step, or stop? Each decision needs different evidence. If the choice is whether to expand, the AI pilot must show what happens at realistic volumes and with normal exceptions, not only on a hand-picked set of easy cases.
Write down what the AI will do, what remains with a person, which cases must be escalated, and what the system must never decide on its own. This small boundary gives the AI pilot a workable test and helps protect customers.
Record the current process before the test
Choose a period that represents ordinary work. Note the number and type of tasks, the people involved, the steps, the handoffs and the exceptions. If demand changes by weekday or season, include that pattern in the comparison or explain why the test period differs.
For the task in the example, a useful starting set might include:
- How many eligible questions arrive, and how many fall outside the agreed scope?
- How long does the full task take, including reading, drafting, checking, correcting and escalation?
- How often does the reply need correction or a second contact?
- How often does the team meet its response promise, and what do customers report?
- Which employee roles spend time on the task, and what other work is delayed?
Keep the definitions simple. If “handling time” excludes review and correction, it will make the old process look slower and the AI pilot look better than it is. Agree who records each measure and where the record lives. Use the minimum customer information needed for the test, and check the applicable privacy and data-handling rules for the markets where the business operates.
Measure the whole result, not just the first draft
An AI pilot can shorten one step while adding work elsewhere. Track the full task from arrival to completion. Include review time, correction, escalation, repeat contacts and any delay caused by a tool or data handoff. A fast suggestion is not a business gain if employees spend longer checking it or customers have to ask again. Measure the complete AI pilot workflow, not just the generated answer.
Set at least one quality measure beside the time measure. Depending on the task, that might be the share of answers that need correction, completion on the first contact, an agreed quality-review result or a customer feedback measure. Decide in advance what level of error or harm requires a pause and a human review. The right measure depends on the task and the customer risk.
Also record the full cost of the AI pilot. Include subscription or usage charges, setup, integration, staff training, ongoing review and the cost of handling failures. Separate cash costs from staff time. Time freed up is capacity; it becomes a financial saving only if the business can actually reduce spending or use that capacity productively.
Compare your AI pilot fairly
Compare similar work. If possible, run the old process and the AI-supported process on comparable cases during the same period. If that is not practical, document the difference in workload, staff, season or case mix. Do not select only easy cases for the AI pilot and then compare them with every case in the baseline.
Keep a simple record for each case or a consistent sample. Mark whether the case was eligible, whether a person changed the AI output, whether it was escalated, and whether a later correction or repeat contact followed. Protect personal and confidential information. A small pilot can still reveal a pattern, but a small or unusual sample cannot establish that the same result will hold at a larger scale.
The OECD’s 2026 D4SME survey covered more than 2,000 small and medium-sized enterprises across 12 OECD countries. The OECD describes the sample as non-representative and reports that secure, targeted integration into business operations remains uneven. That is useful context, not a prediction for every company or country. Read the OECD survey and its methodology.
Agree on a stop, adjust or expand rule
Before the test, write down what would count as a good result, what would trigger a change, and what would stop the AI pilot. For example, an owner might require that total staff time falls without an increase in corrections or repeat contacts. That is an illustrative decision rule, not a universal benchmark.
Give one person responsibility for reviewing results and one person authority to pause the AI pilot if the system produces unsafe or unreliable output. A risk framework can help teams structure that work. NIST’s AI Risk Management Framework is voluntary, and its functions are to govern, map, measure and manage AI risks. It is a planning aid, not a substitute for legal advice or local requirements. See NIST’s AI Risk Management Framework.
At the review, separate what the team observed from what it expects. “The test group needed fewer corrections during this period” is an observation. “A wider rollout will reduce costs” is a forecast that needs further evidence. Keep the assumptions visible, including training time, system limits and the cases that still need human judgment.
Turn the baseline into a better operating decision
A useful AI pilot does not have to prove that AI belongs everywhere. It should help the business choose a sensible next step: stop, redesign the task, improve the data or handoff, extend the test, or expand with clear oversight. That is what makes an AI pilot useful to an owner.
For a small team, a one-page record is often enough: the task and its boundary, the baseline period, the measures, the pilot period, known differences, the cost assumptions, the stop rule and the decision owner. Review it with the people who do the work. They can spot hidden steps and customer risks that a dashboard may miss.
DBG360 can help owners map a process, agree on useful measures, assess people and system handoffs, and turn pilot evidence into a practical implementation plan. If you are weighing an AI test or a broader process change, contact DBG360 to discuss the decision.
