The study highlights the importance of AI oversight to prevent alert fatigue, and the effects of “dataset shift” on widely implemented models.

A descriptive study in JAMA Network Open—involving 24 hospitals across four geographically diverse health systems (the University of Michigan, New York University Langone Health, Mass General Brigham, and BJC Healthcare)—looked at the number of sepsis alerts in the months before and during COVID-19—and found a significant jump in alerts after the start of the pandemic, despite lower patient numbers from canceled elective surgeries.

The Epic Sepsis Model (ESM) is a widely implemented artificial-intelligence- (AI-) based system that will generate an alert if a patient displays a sufficient number of variables indicating a risk of sepsis. “The ESM calculates a score from 0-100, which reflects the probability (or percent chance) of a patient developing sepsis in the next 6 hours,” explains author Karandeep Singh, MD, MMSc, an Assistant Professor of Learning Health Sciences, Internal Medicine, Urology, and Information at U-M; Singh also heads U-M Hospitals’ Clinical Intelligence Committee. “We use an alerting threshold of 6 at Michigan Medicine on the inpatient side, and many other institutions use a similar threshold of 5 or 6 to generate alerts.”

“COVID-19 was a ‘black swan’ event that likely affected many existing models in ways we don’t yet fully understand.”
— Karandeep Singh, MD, MMSc

With an alerting threshold of 6, “our study found that the total volume of daily sepsis alerts increased by 43%, even though hospitals had cancelled elective surgeries and reduced their census by 35% to prepare for the [COVID] surge,” Singh says. One reason for the higher number of alerts despite a lower patient census is that the patients who filled hospitals during COVID’s first wave were, in general, in more critical condition than elective surgery patients and therefore more likely to generate alerts.

Another reason for the jump in alerts could be a result of “dataset shift,” where a model’s performance will deteriorate when there are sudden, unexpected changes in a hospital’s case mix (e.g., a COVID surge). (Singh previously described the phenomenon of dataset shift in a July 2021 New England Journal of Medicine paper.) “COVID-19 was a ‘black swan’ event that likely affected many existing models in ways we don’t yet fully understand,” says Singh. “Future studies that use post-pandemic data to evaluate AI models should be careful to interpret their findings within the context of the pandemic.”

And aside from what caused it, “the increase in total alerts is illuminating because it provides a sense of just how busy front-line clinicians were during the first wave [of COVID], despite the lower hospital census,” he says.

During the first wave of COVID, “at Michigan Medicine, nurses qualitatively reported a rise in sepsis model-driven alerts at levels that felt untenable,” he continues. “Although our study does not quantify alert fatigue, our primary motivation for looking at sepsis scores during this period was that nurses at Michigan Medicine reported a large rise in alerts that ultimately led to these alerts being paused”: a pause that corresponded to an increase in alert volume. Singh explains that the “rise in sepsis scores appears to have been a common finding at many hospitals around the country.”

The study points to the importance of monitoring sudden increases in alert volume, both to prevent alert fatigue for overworked healthcare workers, and to look into whether some form of dataset shift is affecting a model’s accuracy.

“We need to have a way to anticipate and deal with situations where the alert volume is high,” says Singh. “Even among well-resourced hospitals, there is a limit to the capacity to respond to model-driven alerts. Clinical AI governance is a first step toward providing real-time guidance to health systems when situations like this arise, and for setting standards based on local constraints.”