Wrong AI advice can do more than change an answer. It can change the threshold at which a person decides they know enough to answer. A new five-experiment preprint found that access to systematically wrong AI advice sharply reduced people's willingness to say "I don't know," even as accuracy fell and confidence rose. The result is important, but its narrow design matters. The practical lesson is to build independent judgment and verification into the workflow.
What the study tested
The preprint, AI advice suppresses people's willingness to say "I don't know", reports five experiments with 3,132 participants. Four studies were preregistered and one was a direct replication.
Participants answered six difficult questions about visual details in films. They could always decline to answer. Depending on the condition, they had no AI, could request AI advice, or saw AI advice automatically. Some conditions added a small incentive: $0.10 for a correct answer, minus $0.10 for a wrong answer, and no change for declining.
The researchers deliberately selected questions that Step 3.5 Flash usually answered incorrectly. This design isolated what happens when a fluent answer is available but unreliable. It did not estimate the average effect of using modern AI across normal tasks.
The preregistered primary outcome was judgment suspension: choosing not to answer because the participant did not know.
Three results, three different analysis sets
News coverage compressed the study into a striking sequence: judgment suspension fell from 44% to 3%, accuracy fell from about 27% to 9%, and confidence rose from about 30 to 76. Each number has support in the paper, but they do not all come from the same experiment or participant group.
The precise reading is:
- In Study 1b, participants without AI suspended judgment on 44% of questions. Participants who could seek AI advice did so on 3%.
- Across no-incentive conditions pooled by the authors, correctness was 27.5% without AI and 9.2% with AI.
- In Study 2 without incentives, mean reported confidence was 29.6 without AI and 75.9 with AI.
The pattern is still substantial. AI access made participants answer more questions, get fewer right, and feel more certain. Precision about the denominators prevents a media-friendly composite from becoming a fictional single experiment.
The study is about human thresholds, not model confidence
This result is often grouped with research showing that language models can sound confident while being wrong. The paper asks a different question: what does an available AI answer do to a human decision to commit or abstain?
Without AI, uncertainty can produce a pause. A person may leave the answer blank or say "I don't know." A fluent suggestion fills that empty space. Even when the suggestion is wrong, it lowers the friction of committing to an answer.
The authors propose several possible mechanisms, including trust, fluency, lower effort, and altered metacognitive monitoring. Their experiments do not identify which mechanism dominates. They also do not show a permanent decline in critical-thinking ability. They measure short-term answers, confidence, and abstention in a controlled task.
This distinction matters for product design. Improving a model's calibrated confidence can help, but a model saying "80% confident" still provides no independent evidence. Confidence is a routing signal for verification effort. It is not proof.
Small stakes helped, but did not restore the baseline
Study 3 made the incentive visible before every question. In the AI condition, incentives increased judgment suspension from 2% to 8% and correctness from 11% to 16%. Participants also requested AI advice less often, falling from 5.27 to 4.53 requests across six questions.
The intervention changed behavior. It did not restore the no-AI baseline.
This supports a useful product principle: error cost must be visible at the decision point. Interfaces that optimize only for speed and seamless completion make commitment cheap. A small amount of structured friction can make people inspect the recommendation more carefully.
The study did not test warning labels, mandatory citations, delayed recommendations, or a formal verification workflow. Those interventions remain design hypotheses.
Where the evidence stops
Five limitations should govern any application of the findings.
First, all questions concerned minor visual details in films. The domain was intentionally trivial and difficult.
Second, the AI was selected because it was almost always wrong on these questions. The effect when AI is usually correct remains untested.
Third, the monetary stakes were small. Ten cents per answer does not represent a medical diagnosis, a legal filing, an investment decision, or professional accountability.
Fourth, the researchers did not record whether participants checked the web, consulted another model, or used another source. Improved correctness under incentives may reflect lower AI reliance, more verification, or both.
Fifth, this is a preprint. Its design includes preregistration and direct replication, but peer review is still pending.
The right conclusion is specific: when people face questions they do not know and receive fluent, systematically unreliable AI advice, their willingness to abstain can collapse. The study does not establish that ordinary AI use reduces accuracy by a factor of three.
Related research points in the same direction
A 2025 CHI study from Microsoft Research surveyed 319 knowledge workers about critical thinking during GenAI-assisted work. Higher confidence in AI was associated with less reported critical thinking, while higher confidence in one's own ability was associated with more. Participants described verification, integration, and task stewardship as important parts of the work that remained.
The Microsoft study is observational and self-reported, so it cannot establish causality. It still provides a useful real-work complement to the controlled movie-question experiments.
Research on model calibration addresses another layer. The RLCR paper trains a model to produce better-calibrated confidence estimates. Better calibration can make uncertainty more observable. It does not automatically prevent a user from over-weighting a fluent answer, and it does not replace citations or external checks.
The combined evidence suggests three separate engineering targets:
- Model calibration: can the model expose uncertainty?
- Human calibration: can the user preserve the option to abstain?
- System verification: can the workflow detect whether the claim is true?
Reliable products need all three.
A three-stage verification loop
The following workflow is a design inference from the evidence, not an intervention tested by the preprint.
Stage 1: Commit before consulting
Before showing AI advice, capture the user's initial answer, rationale, confidence, and an explicit "I don't know" option. Freeze that snapshot.
This creates a baseline. It prevents the AI answer from silently replacing the user's first judgment and makes later movement visible.
For low-risk tasks, the commitment can be lightweight: one sentence or a confidence choice. For high-risk decisions, require a reason and relevant evidence.
Stage 2: Compare, rather than overwrite
Show the AI answer with its assumptions, source links, and uncertainty. Generate a diff between the initial judgment and the recommendation.
Ask the user to mark each conflict as accepted, rejected, or requiring verification. The interface should preserve both versions instead of letting the AI answer become the only visible state.
This shifts interaction from passive acceptance to explicit adjudication.
Stage 3: Verify through an independent channel
High-risk claims need a check independent of the generating model. Depending on the task, that can be:
- A primary source or authoritative database.
- A deterministic calculation.
- A unit test, simulation, or reproducible query.
- A second expert with independent context.
- A separately isolated model using different evidence.
Save the verification artifact with the final decision. The artifact can be a URL, query result, test output, signed approval, or calculation sheet. This creates an audit trail and makes errors easier to diagnose.
Four product controls worth testing
The loop translates into four concrete controls.
Delay AI advice until an initial judgment exists for tasks where anchoring matters. Make "I don't know" a first-class successful outcome. Use confidence to set verification depth, never as a substitute for evidence. Finally, apply risk tiers: low-risk work can accept fast review, while medical, legal, financial, security, and production changes require independent evidence or a second-person gate.
This aligns with a larger principle of trustworthy AI systems: trust comes from verifiability. A persuasive answer can be useful, but the system earns trust by making errors detectable.
FAQ
Does AI advice always reduce human accuracy?
No. This study deliberately used questions on which the selected model was usually wrong. It isolates the behavioral effect of unreliable advice and does not measure the average value of AI assistance.
Why is asking the AI for a confidence score insufficient?
Self-reported confidence is another model output. It can help route uncertain cases, but a high score does not independently verify the underlying claim.
Did money stop participants from following wrong AI advice?
Small incentives reduced AI use and improved accuracy, but they did not restore judgment suspension to the no-AI level.
Would a model that says "I don't know" solve automation bias?
It would improve one signal. Users can still over-trust confident answers, and the model can still be miscalibrated. Independent evidence and workflow controls remain necessary.
Which AI outputs require independent verification?
Verify outputs when an error can change money, rights, health, security, production systems, or public claims. Also verify precise numbers, recent events, and claims that conflict with domain expertise.
References
- Chiara Marcoccia, Walter Quattrociocchi, and Valerio Capraro. AI advice suppresses people's willingness to say "I don't know", 2026 preprint.
- Hao-Ping Lee et al. The Impact of Generative AI on Critical Thinking, CHI 2025.
- Zhenlin Xu et al. Reinforcement Learning for Confidence Calibration, 2025.
- Parasuraman and Manzey. Complacency and Bias in Human Use of Automation, 2010.