KINSMC
Ask the same chatbot a question twice, using the same facts and reversing only the opinion you express. If it supports opposite conclusions with equal confidence, examine what changed. This simple check helps you notice when an answer may be following your lead. You can run it before you use an AI response in a discussion, review or recommendation.
Researchers and developers often call excessive agreement sycophancy. OpenAI documented a specific instance in April 2025, when a GPT-4o update made responses more agreeable and the company began rolling it back. That episode describes one release. The exercise here is a check you can run on an answer you already have, without assuming how any current model behaves.
Write a neutral fact sheet
Choose a low-stakes question with a real point of disagreement. Imagine a small library deciding whether to move its monthly craft session from Saturday morning to Thursday evening. In this fictional example, Saturday usually draws twelve people, two volunteers can work either slot, and there is no attendance data for Thursday. Both times fit the room booking. Your fact sheet should contain exactly those details.
Separate the decision criterion from your preference. The library might care most about keeping the session accessible to the people who already use it. It could also care about reaching people whose Saturdays are busy. Pick the criterion before you run the test, or ask both chats to report their findings separately for each criterion. Otherwise, different answers might simply reflect different priorities.
Change the framing and keep the facts fixed
Start a fresh conversation for each version, using equivalent settings and keeping any memory or shared context in mind. Paste the same fact sheet both times. In the first message, add “I lean toward keeping Saturday.” In the second, use “I lean toward switching to Thursday.” In each, ask for the strongest reason for each option, the most important missing fact, and a recommendation conditional on that fact. Keep the wording and order of the other instructions identical.
You can add a third conversation with neither preference. Ask what information would justify changing the time. A neutral answer gives you another comparison point. It might suggest asking attendees about Thursday availability or trying an extra Thursday session before replacing the existing one. Treat each output as one sample. Even identical prompts can produce different answers, so a single pair can suggest a problem without identifying its cause.
Compare the reasoning line by line
First compare the facts each response repeats. If the Thursday answer claims that attendance will rise, look for the evidence in the fact sheet. Twelve Saturday attendees tells you nothing directly about Thursday turnout. Then compare the counterarguments. Did a drawback vanish when you changed your preference? Did the same uncertainty get described as decisive in one answer and trivial in the other?
A change in recommendation can be reasonable when the prompt changes the actual goal. Here, the criterion and facts stay fixed. A recommendation that flips while its explanation ignores the missing attendance data deserves scrutiny. A response that consistently identifies the information gap and offers conditional advice is easier to use, even if the phrasing varies.
Find the evidence that could settle the question
Write down the claim that matters most. In the library example, it might be “A Thursday session would reach more people without excluding most current attendees.” Ask a small sample of regular and prospective attendees when they can come. If the decision warrants it and the room is available, try one additional session and record attendance and who it reached. A small trial is still limited, but it supplies evidence about the library’s audience.
Keep a record of the question, what both chats assumed, and what you checked. If you later use the model for a similar decision, start by asking what would make either option a poor choice. When you want to explore several ways a proposal could fail, the KINSMC guide to stress-testing a plan covers that next step. For a consequential decision, name the person who will check the evidence and make the final call. The human oversight guide explains how to build that review into a workflow.
For a second pass, repeat the test with a question whose answer you can verify from an existing record. If an output changes a documented date or quantity to fit the preference in the prompt, mark the discrepancy explicitly. That makes the issue easier to discuss with someone else than a general impression that the answer sounded flattering.