An artificial intelligence tool detected brain aneurysms that radiologists missed in routine clinical practice, increasing the number of identified cases by 39%, but it also produced substantially more false-positive findings than radiologists.
The findings, published in the Journal of the American College of Radiology, suggest that AI could serve as a second set of eyes for radiologists, while highlighting limitations that could affect its value in different clinical settings.
Researchers at Northwell Health evaluated an FDA-cleared algorithm from Aidoc designed to identify suspected intracranial aneurysms on CT angiography (CTA). The deep learning software analyzes scans for findings suggestive of an aneurysm and flags suspicious cases for radiologist attention.
The study included 3,856 CTA examinations across a large health system. The AI operated in “shadow mode,” meaning it processed scans without showing its results to radiologists or influencing patient care. Independent neuroradiologists reviewed cases in which the AI and clinical interpretations disagreed.
Radiologists and AI agreed on more than 96% of examinations, but each caught aneurysms the other missed. AI detected 55 confirmed aneurysms that radiologists had not reported, increasing the number of cases found by 39% compared with radiologists alone. Radiologists, however, detected 30 confirmed aneurysms that AI missed.
The AI’s greater sensitivity came with an important tradeoff. It detected 84.6% of true aneurysms compared with 71.8% for radiologists, but its positive predictive value was considerably lower—78.2% versus 92.7%. In other words, more than one in five aneurysms flagged by AI were not confirmed as true aneurysms.
The limitation was even more apparent among findings identified only by AI. Of 101 AI-only findings, 55 were confirmed aneurysms and 46 were false positives.
AI performance also varied by clinical setting. Among inpatients, the algorithm added 18 true aneurysm detections while producing seven false-positive alerts. Results were also favorable in the emergency department. In outpatient care, however, AI added only four true detections and generated more false-positive than true-positive findings.
Many of the additional aneurysms detected by AI were small, which could provide opportunities for risk assessment and monitoring before rupture.
Overall, the findings suggest that AI may help radiologists catch additional aneurysms, but its added sensitivity comes at the cost of false-positive findings requiring physician review. The differences across care settings also underscore the importance of evaluating medical AI in the clinical environments where it will actually be used.
