Researchers determine that AI-based tools have not yet reached full diagnostic potential in COVID-19

published by the magazine Radiology: Artificial Intelligencea prospective observational study across 12 hospital systems at the University of Minnesota Medical School evaluated the real-time performance of an interpretable artificial intelligence (AI) model for detecting COVID-19 from chest radiographs.

Participants with COVID-19 had a significantly higher COVID-19 diagnosis score than participants who did not have COVID-19. However, the researchers found that the performance of the real-time model did not change during the 19 weeks of implementation. The sensitivity of the model was significantly higher in men, while the specificity of the model was significantly higher in women. Sensitivity was significantly higher for Asian and black participants than for white participants. The AI ​​of COVID-19 diagnostic system it had significantly worse accuracy than the predictions made by radiologists.

“This study, which represents the first in vivo investigation of a COVID-19 AI diagnostic model, highlights the potential benefits but also the limitations of AI,” said Christopher Tignanelli, MD, MS, FACS, FAMIA, associate professor of surgery at the University of Minnesota Medical School and a general surgeon at M Health Fairview. “While promising, AI-based tools have not yet reached their full diagnostic potential.”

The research findings were informed by an AI algorithm developed by Ju Sun, an assistant professor in the U of M College of Science and Engineering, and his team in collaboration with M Health Fairview and Epic.

• COVID-19 diagnostic models work well for participants with severe effects of COVID-19; however, they fail to differentiate participants with mild effects of COVID-19.
• Many of the early pandemic AI models that were published boasted overly optimistic performance metrics using publicly available data sets.
• The diagnostic accuracy of the AI ​​model was lower than the predictions performed by board-certified radiologists.

“We observed the same overly optimistic performance in this study when we validated against two publicly available data sets; however, as we show in our manuscript, this does not translate to the real world,” Dr. Tignanelli said. “It is imperative to move forward that both researchers and journals develop standards that require real-time or external prospective validation for peer-reviewed AI manuscripts.”

The researchers hope to develop a simpler diagnostic AI model by integrating data from more than 40 US and European sites and multi-modal models that leverage structured data and clinical notes alongside images.