KINSMC
An AI prompt can work well for weeks and then change because someone adds a line that solves one annoying problem. Imagine a small museum using a shared prompt to turn exhibit notes into draft listings. The team asks for shorter listings, and the new draft fits the website. It also drops the line telling wheelchair users to enter through the courtyard. The edit improved length while breaking a detail the team needed to keep.
A check before rollout gives the museum a way to catch that kind of regression, a behavior that worked before a change and fails afterward. The comparison starts with the prompt already in use and a small set of cases the team can rerun. If you are still deciding whether to adopt AI for the work, start with our guide to testing an AI workflow before using it at work. The steps here apply after the team has a working version to compare against.
Save the version your team is using
Copy the current instructions into a shared document with a version label, a date and the name of the person who maintains them. Record relevant settings and any reference material that the prompt depends on. If the tool displays a model identifier, record it. If it exposes limited details, write down what you can see and when you checked it. This helps the next reviewer judge what changed.
Then edit a copy. In the museum example the change is a request to reduce the listing to 80 words. Keep the requirements for time, location, access directions and booking details visible in both versions. Write down the intended result before running the new prompt. A colleague should be able to tell what improvement you are testing from a single sentence.
Save examples that cover the details you cannot lose
Start with a routine case, such as a weekday exhibit at the main entrance. Add a case with a different entrance, an event that moved rooms and an event with no booking link supplied. Use approved examples or make fictional equivalents with the same features. Save the full input for each case, including the notes or instructions that would be visible when doing the actual task.
Give each example an answer key for the fields that matter. For the courtyard example, the key might require the date to match the exhibit notes, the access directions to be present and the copy to fit within 80 words. Record what to do with missing information. For a listing with no booking link, a passing draft can leave a question for the editor. A link invented by the tool fails that check, regardless of how clearly the draft reads.
Compare both versions on the same inputs
Run the old and revised prompts separately using the same saved input. Start each in a fresh session when that matches how the team works, and keep other settings fixed where possible. Save both outputs before editing them. Check the named fields first and write down what passed or failed. If the old prompt already drops the courtyard directions, the comparison has uncovered an existing defect. If only the revised version drops them, you have evidence of a regression on that case.
If a case matters enough to block rollout, try it more than once. A useful record says how many times you ran it and what varied. The NIST report on monitoring deployed AI systems discusses variation in outputs and inputs as reasons to monitor real use. A handful of repeated cases can expose a problem without establishing a reliable failure rate for every listing. The NIST Generative AI Profile provides further context on evaluation and monitoring.
Decide who can switch versions and when to switch back
Use a pass rule the reviewer can apply without guessing your intent. For the museum, any missing access direction or wrong event date blocks the switch in the saved cases. When the revised version passes, try it with one editor and check the first few real listings before extending the change. Keep the previous instructions easy to retrieve. If the team cannot safely use either version, follow the manual fallback procedure for an AI workflow while investigating.
When a reviewer catches a new failure in live work, capture a shareable version of the input and add it to the saved cases. Give it an expected result and record which prompt versions were affected. The next time the team asks to shorten a listing, translate it or change the format, the reviewer can check that same access detail before anyone publishes a draft.