Business Direction
Write down the baseline before the pilot starts
An AI pilot proves nothing without a before number. Four measurements, taken in an afternoon, turn “people liked it” into a decision.
By Paul Engerman ·
Before an AI pilot begins, write down four numbers for the task it is meant to help: how long the task takes, how often it happens, how often it has to be redone, and what that adds up to in hours. Take the same four measurements at the end. Without the first set, the second set cannot tell you anything.
That holds whether you lead a growing company or a department in a municipal or county government. The licenses cost real money, and sooner or later someone will ask what they bought.
Why “people liked it” is not a result
Most pilots end the same way. A few enthusiasts love the tool, a survey comes back mostly positive, and the pilot is declared a success. Then the renewal arrives, and someone in finance or on the council asks what changed. Nobody can say.
Liking a tool and saving time are different things. Some people enjoy a tool that saves them nothing. Others save hours and still grumble. A survey measures the first. Only a before-and-after number measures the second.
Four numbers to write down first
Pick the one task the pilot is for. If you have not named it yet, start with naming the problem. Then measure it as it runs today.
| Measure | How to get it | Example (illustrative) |
|---|---|---|
| Time per task | Time five real instances with a phone timer | About 40 minutes to draft a monthly vendor summary |
| Volume | Count it from the system, the inbox, or the log | 12 summaries a month |
| Rework | Count how many go back for correction | 3 of 12 returned last quarter |
| Hours | Time per task × volume, plus rework | About 8 hours a month, before rework |
Add one sentence about quality: what a good-enough result looks like, and who decides. Date the page and keep it.
Rough is fine. You are not publishing research. You need a starting point honest enough that a later number means something.
Run the pilot like a test
- Use a small group of five to ten people who do the task every week. If you can, keep a few people doing it the old way for comparison.
- Give it a fixed length. Thirty days is enough for most tasks. Expect the first week to be slower while people learn, and judge on the last two weeks.
- Measure the same way at the end. Same task, same timer, same count, same rework check.
- Write the decision rule before you start. For example: “If time per summary drops by a quarter and rework does not rise, we extend it to the whole team.” Deciding the bar in advance keeps the result from being argued into whatever anyone hoped for.
What the built-in reports can and cannot tell you
Microsoft 365 includes reports that help, but they answer a different question. The Copilot usage report in the admin center shows how many licensed users were active and how many prompts they submitted. That tells you whether people used the tool, not whether it paid off.
The Copilot Dashboard in Viva Insights goes further and estimates “assisted hours.” It counts Copilot actions and multiplies them by factors from Microsoft’s research, for example six minutes for each search or summary action, based on a study of 163 knowledge workers. It then puts a dollar value on those hours at a default rate of $72. Microsoft itself calls these broad approximations rather than precise calculations.
Use these reports to see who is using the tool and where adoption is stalling. Do not let them stand in for your own before-and-after measurement of your own task.
When the numbers say no
A pilot that shows no gain is a useful result. It just saved you the cost of a full rollout.
When the numbers disappoint, the cause is usually upstream of the tool. The assistant could not find or trust the right files, which is an information architecture problem. Or the task was a poor fit, or nobody showed staff how to use it. Fix the cause and test again, or stop and spend the money elsewhere. Either way, you decided from evidence.
Where this shows up in the assessment
This is sub-score 2.3, measurement and return, in the Business Direction pillar, with 2.2, sponsorship and budget, close behind. A low grade usually means licenses were bought before anyone agreed how to judge them.
A short workshop can fix it. It picks two or three tasks, takes the baseline, and sets the decision rule before the pilot starts. If you want to see how your organization would grade, read how the assessment works.
Sources
- Microsoft Copilot usage report (Microsoft Learn)
- Connect to the Microsoft Copilot Dashboard for Microsoft 365 customers (Microsoft Learn)