Better answers with AI do not prove we can spot its mistakes

A radiologist can get more diagnoses right with an AI assistant and still follow it when it is wrong. That matters when we use these tools in training. A resident’s improved score may tell us that the assistant was useful, but it does not tell us whether the resident has learned to judge its advice.

A recent study in Radiology measured both sides of that problem. Readers answered more cases correctly with LLM assistance. But those who performed less well on their own also accepted more incorrect suggestions from the model. Better explanations were associated with greater acceptance of advice, including advice that was wrong.

The study did not test whether AI helps or harms learning over time. It did show why a training program needs to assess what a resident can do independently, even when their work with AI looks better.

What the study measured

Song and colleagues asked ten readers, including thoracic radiologists, general radiologists, and residents, to interpret 100 chest imaging cases selected from a teaching collection. Each case offered five possible diagnoses. Readers first answered on their own, then reviewed the cases with the model’s suggested diagnoses, confidence scores, and explanations.

The researchers used two assistants to provide advice of different accuracy. GPT-5 received clinical information and descriptions of the imaging findings written by radiologists. GPT-4o received the clinical information and the images themselves. Their first-choice diagnoses were correct in 76% and 27% of the assigned readings, respectively.

That difference needs care. Both the model and the information it received changed, so we cannot tell how much of the difference came from the model and how much came from having a radiologist describe the findings first.

The main outcome was whether readers accepted correct suggestions and rejected incorrect ones. This adds something that overall accuracy cannot show. An assistant can correct enough mistakes to raise a reader’s score while also introducing new mistakes when the reader follows bad advice.

Two thoracic radiologists rated the explanations on a five-point scale for factual grounding and clinical logic. They scored the quality of the explanation separately from whether the proposed diagnosis was correct. A model could therefore receive a good explanation score while giving the wrong diagnosis.

Each one-point increase in explanation quality was associated with higher odds of accepting incorrect advice, adjusted OR 1.71 (95% CI 1.47–1.99). Better explanations were also associated with lower odds of rejecting correct advice, OR 0.79 (95% CI 0.67–0.93). In other words, readers were more likely to accept suggestions accompanied by better explanations, whether the diagnosis was right or wrong

.

Each one-point increase in explanation quality was associated with fewer rejections of correct advice and more acceptance of incorrect advice. These are associations within the reader study. Source: Song et al., Table 3.

The researchers did not randomly assign the quality of the explanations, so this does not prove that improving an explanation causes readers to make more mistakes. It does show that a well-constructed explanation can accompany a wrong answer and fail to help the reader recognize the error.

What the abstract hides

The authors defined expertise by how accurately each reader answered the cases without AI. They did not define it by job title or years of experience. The higher-performing group included three thoracic radiologists and two residents; the lower-performing group included two general radiologists and three residents. Calling this a comparison of attendings and trainees would misrepresent the study.

Readers in the lower-performing group accepted 45.8% of the model’s incorrect suggestions, compared with 30.4% in the higher-performing group. Yet they also gained more from assistance: their overall accuracy rose by 16.8 percentage points, compared with 9.4 points in the higher-performing group. They benefited more from the tool while also being more likely to follow it into an error.

Five readers per group, grouped by unaided accuracy. Each reader interpreted the same 100 cases. The counts describe their decisions, not outcomes in separate patients. Source: Song et al., Table 4.

The higher-performing readers did not appear to reject useful advice more often. Rejection of correct suggestions was 14.1% in the lower-performing group and 15.6% in the higher-performing group, with no statistically significant difference. Their advantage was in recognizing more of the wrong suggestions, rather than disagreeing with AI more often across the board.

There is also a limit to what this measure tells us. A reader who rejects the model’s wrong diagnosis can still choose a different wrong diagnosis. We need both measures: how readers respond to advice and whether they ultimately reach the correct answer.

The authors caution that ten readers working through difficult, multiple-choice teaching cases may behave differently from radiologists in routine practice. The public cases may also have appeared in the models’ training data. Most relevant to this essay, the study did not follow readers over time to see whether their own skills improved or deteriorated.

This also connects to the previous issue, Tested against physicians. The more accurate assistant relied on findings that a radiologist had already identified and described. Some of the expertise supporting its answers was supplied before the model began. The same needs to be acknowledged when we evaluate AI that works from radiology reports.

When explanations helped readers

Pesapane and colleagues found that explanations helped in a different setting. Six breast radiologists read 200 mammograms. To test how they responded to misleading advice, the researchers deliberately shifted the AI’s BI-RADS-like recommendation by one category in 30% of examinations.

Among those altered recommendations shown before the reader’s own assessment, acceptance of the misleading advice fell from 36.1% without explanations to 17.8% with saliency heatmaps, adjusted OR 0.56 (95% CI 0.44–0.71). These heatmaps highlighted image regions contributing to the AI output.

This is a useful counterpoint to Song, but it does not prove that heatmaps are safer than written explanations. The studies used different tasks and readers. Pesapane’s team also changed the diagnostic category while leaving the original heatmap unchanged. Their experiment tested a particular kind of misleading output, and the authors caution against treating its bias rates as estimates for routine screening.

The practical question is whether an explanation helps readers recognize errors in the setting where it will be used. These two studies give different answers under different conditions.

The concern about never-skilling

These studies suggest that what a reader already knows affects how well they can use AI advice. That creates a question for training: if we rely on knowledge to help readers detect AI errors, how do we make sure trainees acquire that knowledge while using the assistant?

A dermatology study by Xu and colleagues offers limited support for this concern. They compared 320 medical students with 153 primary care physicians on the same image-based differential diagnosis task. Students were more likely to get cases right only when the AI was right, while incorrect advice had little measurable effect on physicians’ accuracy. This suggests that existing clinical knowledge may help readers evaluate AI advice. It does not establish the same effect in radiology or show whether using AI changes what trainees learn.

Deskilling means losing proficiency in something we once knew how to do. Never-skilling is the concern that someone may not fully learn it in the first place. Neither has been demonstrated in radiology by these studies. What they show is that better answers with assistance can coexist with difficulty rejecting bad advice.

I am concerned about a resident who routinely sees the proposed diagnosis before working through the case. Their final answer may be correct, but I may not know whether they identified the relevant findings, considered alternatives, or understood why one diagnosis fits better than another. To assess that, I need to see some of their work before the model supplies an answer.

Aviation provides a reason to take this concern seriously, although it cannot tell us what will happen in radiology. Casner and colleagues studied 16 airline pilots in a Boeing 747-400 simulator. Instrument scanning and manual control were largely intact. Pilots had more difficulty tracking their position, deciding which navigational step came next, and resolving instrument failures. Those who stayed more mentally engaged while automation was operating tended to perform better on cognitive tasks.

The relevant concern is that watching a system work may not provide enough practice in the skills needed when it fails. We should not assume that radiologists will show the same pattern as pilots, but we can test whether continued AI use leaves them able to interpret unfamiliar cases and recognize misleading advice on their own.

What I’d tell the fellow

I’d want to see an initial interpretation before reviewing the assistant’s answer: the relevant findings, the leading diagnosis, and the feature that makes the closest alternative less likely. That gives us something concrete to discuss when the model disagrees.

I’d then check its explanation against the images and clinical information. Does the finding it names actually exist? Does that finding help distinguish the proposed diagnosis from the alternatives? Has it overlooked something that argues against its answer? Song’s study shows why this matters: an explanation can be largely sound and still support the wrong diagnosis.

I would review cases where the assistant corrected a mistake alongside cases where it changed a correct answer into an incorrect one. The aim is to understand why the reader accepted or rejected the advice, and whether that decision was justified by the case.

What I’d change Monday

For a training program, I’d keep some interpretation unaided, give feedback on it, and assess progress over time. I’d want to know whether trainees become better at unfamiliar cases and whether they can recognize plausible AI errors without first being shown the correct answer. These papers give reasons to measure those abilities; they have not established the best way to teach or preserve them.

For a clinical workflow, I’d consider testing whether recording an initial impression before showing AI advice changes decisions, accuracy, or reading time. Pesapane’s authors propose this approach as a potentially safer workflow. It will not prevent every error: Song’s readers had already answered on their own and still accepted incorrect suggestions.

This is also a design question in RadReason, the radiology reasoning tool I’m building. In Learn Mode, learners describe their findings and work through questions before reviewing the model’s differential. The aim is to make their reasoning visible so it can be discussed. Whether that approach improves their ability to reason independently still needs to be tested.

AI’s benefits matter. Both groups in Song’s study answered more cases correctly with help. I’d want a training program to use that benefit while also showing that the resident is learning to make and explain a judgment independently. I’d be reassured by improvement over time in both unaided interpretation and the ability to recognize and explain errors in the assistant’s advice.

Journal Club Card

Paper: Song J, Jeong WG, Han DH, et al. Determinants of Reader-LLM Interaction in Thoracic Radiology: Impact of Model Confidence and Reader Expertise. Radiology. 2026;320(2):e253397. Full citation.

Conclusion: Both groups answered more cases correctly with AI. Readers who performed less well on their own also accepted more incorrect suggestions. The study did not test whether anyone learned or lost skills over time.

Three limitations

1. Ten readers interpreted 100 multiple-choice teaching cases. Having each reader assess many cases does not tell us how a larger, more varied group of readers would perform.

2. The two assistants differed in both model and input. The researchers did not randomly assign explanation quality, so its association with reader behavior does not establish cause and effect.

3. Expertise meant accuracy on these cases, not seniority. Readers were not followed over time to assess changes in their independent ability.

Two discussion questions

1. If a trainee’s score improves with AI, how would we check whether they can also recognize its mistakes?

2. What would we measure over time to determine whether the trainee is becoming a better reader independently?

Use this for your next journal club.