Impact of different mammography systems on artificial intelligence performance in breast cancer screening
Abstract
Artificial intelligence (AI) tools may assist breast screening mammography programs, but limited evidence supports their generalizability to new settings. This retrospective study used a 3-year dataset (April 1, 2016-March 31, 2019) from a U.K. regional screening program. The performance of a commercially available breast screening AI algorithm was assessed with a prespecified and site-specific decision threshold to evaluate whether its performance was transferable to a new clinical site. The dataset consisted of women (aged approximately 50-70 years) who attended routine screening, excluding self-referrals, those with complex physical requirements, those who had undergone a previous mastectomy, and those who underwent screening that had technical recalls or did not have the four standard image views. In total, 55916 screening attendees (mean age, 60 years ± 6 [SD]) met the inclusion criteria. The prespecified threshold resulted in high recall rates (48.3%, 21929 of 45444), which reduced to 13.0% (5896 of 45444) following threshold calibration, closer to the observed service level (5.0%, 2774 of 55916). Recall rates also increased approximately threefold following a software upgrade on the mammography equipment, requiring per-software version thresholds. Using software-specific thresholds, the AI algorithm would have recalled 277 of 303 (91.4%) screen-detected cancers and 47 of 138 (34.1%) interval cancers. AI performance and thresholds should be validated for new clinical settings before deployment, while quality assurance systems should monitor AI performance for consistency.
Citation
de Vries , C F , Colosimo , S J , Staff , R T , Dymiter , J A , Yearsley , J , Dinneen , D , Boyle , M , Harrison , D J , Anderson , L A , Lip , G , Black , C , Murray , A D , Wilde , K , Blackwood , J D , Butterly , C , Zurowski , J , Eilbeck , J & McSkimming , C 2023 , ' Impact of different mammography systems on artificial intelligence performance in breast cancer screening ' , Radiology: Artificial Intelligence , vol. 5 , no. 3 , e220146 . https://doi.org/10.1148/ryai.220146
Publication
Radiology: Artificial Intelligence
Status
Peer reviewed
Type
Journal article
Rights
Copyright © 2023 by the Radiological Society of North America, Inc. This work has been made available online in accordance with publisher policies or with permission. This accepted manuscript is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. This is the author created accepted manuscript following peer review and may differ slightly from the final published version. The final published version of this work is available at https://doi.org/10.1148/ryai.220146
Description
Funding: Supported by the Industrical Centre of Artificial Intelligence Research in Digital Diagnosis (iCAIRD), which is funded by Innovate UK on behalf of UK Research and Innovation (UKRI) (project no. 104690).Collections
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.