KINSMC
Before you put AI into a recurring task, run the old and new versions side by side. Give both versions comparable inputs, judge them using the same rules, and count the time it takes to correct mistakes. You want to know whether an ordinary colleague can get a usable result on an ordinary day, including the days when the inputs arrive in a mess.
A field experiment with management consultants, published in Organization Science, found that AI assistance improved performance on some tasks and reduced it on another. The study concerned specific tasks, people and tools. Its practical implication is to test your own workflow at the level of an individual task. The pilot below is an editorial method you can adapt, separate from the study’s experimental protocol.
Pick one task with a clear finish line
“Use AI for operations” is too broad to test. Try “turn completed event request forms into draft setup checklists.” The inputs are forms and a current venue guide. The output is a draft with room, equipment, access and timing fields. A named employee still checks the checklist before anyone acts on it. Keep the tool away from the calendar, purchasing system and outgoing mail during the trial.
Write down what a successful draft must contain. For the venue example, room and start time must agree with the form. Any absent detail must be marked as missing. A conflict between two documents should go to a person who can resolve it. Treat a guessed access instruction as a critical error, even if the rest of the checklist looks tidy. A colleague should be able to apply the rules without asking what you meant.
Build a small test set that includes awkward cases
Use cleared or fictional examples, and remove personal or sensitive information. Start with ten cases if that fits your workload. Include routine forms, an incomplete form, a conflicting date, a late change and a case that should be sent back for clarification. Preserve the original formatting quirks when they affect the task. A pilot of perfectly typed forms tells you little about a folder full of scanned amendments.
Make an answer key before you see the AI drafts. Record the expected room and start time, any missing fields, and the cases where the correct action is to stop. Put two or three additional cases aside. You can use them after changing the instructions to see whether you fixed the general problem or just taught the tool to pass examples you already saw.
Measure the whole job, including the review
Run the manual version and record the time from reading the input to accepting the draft. Use that same start and finish for the AI-assisted version. Include preparation, prompting, reading, verification and repairs. Keep setup time in a separate column so you can see how long it would take to recover that investment. If one person runs both versions, switch the order across cases or use matched pairs. The second attempt can benefit from remembering the first.
Here is a fictional timing example. The manual draft took twelve minutes. The AI produced a draft in two minutes, but setup took one minute, checking took six, and fixing omissions took four. The total was thirteen minutes. The automatic first draft looked fast because the timer stopped at the easiest point. Use the median and keep a note of the slowest case so a rare, expensive failure stays visible.
Record the defects a reviewer would need to catch
Use one row per case. Record the method, total minutes, factual errors, omitted details, and whether the draft passed the rubric. Ask a second colleague to score outputs without seeing which method produced them if you can. Style can influence judgments. In a setup checklist, an invented door code matters far more than a clumsy heading.
Decide how to respond before you roll out. A sample with a critical error might justify limiting AI to layout and wording while an employee fills verified fields. If the hard cases pass but editing consumes the time saved, keep the manual process for now. If the workflow saves time and meets the agreed standard, start with a small group, a named reviewer and an easy way to return to the manual process. Save the rubric and rerun it after changing the tool, instructions or source documents.
Make the decision easy to revisit
Write a short decision record with the task, sample date, tool configuration, accepted scope and known failures. Note who will review the first live cases and when to pause. The result applies to the task you actually tried. A successful checklist pilot gives you a reason to keep testing checklist drafts. Expanding it to email replies or purchasing requires a separate trial with different consequences and a different answer key.
Related KINSMC guides cover designing human review and checking how an AI tool handles your data.
Source
Dell’Acqua et al., Organization Science (2026)