Do predictive analytics work in healthcare?

Artificial intelligence has been touted as a healthcare analytics cure-all that will impose order on massive data streams, extract essential insights, and generate predictions with pinpoint accuracy. But is there evidence that AI and predictive analytics have begun to deliver real value?

Although there is good reason to be wary of the hype, recent research suggests that predictive models, when used correctly, can play a powerful role in guiding value-based healthcare delivery and payment. Predictive models output risk scores that estimate the probability of some future event, such as hospitalization, or the predicted value of some future outcome, such as healthcare spending. These scores allow providers to target preventive care to the patients who need it the most, which can lead to improved health outcomes and reduced costs.

Cause for concern

Assessments of healthcare AI have produced a host of worrisome findings:

• An evaluation of AI tools designed to diagnose and triage COVID-19 patients concluded that none were fit for clinical use.
• Reviews of models across medical specialties, as well as those specifically from oncology, obstetrics, psychiatry, and orthopedics, found that most were at high risk of bias, meaning that model predictions were likely to be worse than expected when applied to new patients.
• Previous AI testing has also failed to consider that these systems perform differently in experiments than they do in the clinic.
• An additional concern is whether predictive models treat all patients fairly.

Given these myriad issues, it is unsurprising that most models are never used in practice.

Promising results

Although healthcare AI has a lengthy record of miscues, recently published population health research provides reason for optimism. One example comes from Independence Blue Cross of Philadelphia, a health insurer that uses a predictive algorithm to assign risk scores to its congestive heart failure patients. The risk scores help determine the priority for a preventive intervention, with highest-risk members receiving telephone outreach to help them manage their health conditions.¹ The goal of the program is to reduce acute care utilization in the form of cardiologist visits, emergency department visits, and hospitalizations.

The intervention was applied in ten waves over roughly two years. For each wave, outreach was determined not only by a patient’s absolute level of risk but by the capacity of the care coordination team, which varied over time. These arbitrary cutoffs created a type of quasi-random assignment that researchers could exploit to estimate the program’s causal efficacy with a regression discontinuity approach.

Researchers credit the program with producing large decreases in the risk of cardiology and emergency department (ED) visits, with about a 20% relative reduction in each compared with the control group over one year. The decrease in the likelihood of 180-day ED visits was even larger, with a 38% reduction relative to the control group. The authors note that the observed reduction in ED visits would imply a savings of $500 to $1,000 for ED visits per member per year. The treatment group also displayed a decline in hospitalizations, although that effect was not statistically significant. Nonetheless, the effect on hospitalizations would imply an annual savings of about $13,000 per member. The authors hypothesize that these large effects may be due to successfully targeting the sickest members who have the most to gain from the intervention.

Another pair of recent studies evaluated how an intervention influenced the healthcare utilization and costs of older patients at the Mass General Brigham integrated delivery system. Eligible patients² were randomly assigned either to a control group, in which they received care as usual, or to the “Stepped-Care” intervention program composed of predictive analytics plus nurse-driven interventions. Mass General Brigham applied a predictive model that estimated each patient’s risk for 30-day emergency hospital transport. Stepped-Care patients were then stratified by their risk scores, and high-risk patients were triaged into care plans tailored to their needs.

Researchers found that patients in the intervention group had 68% fewer 90-day readmissions and 53% fewer 180-day readmissions relative to the control group, both significant improvements. In addition, the intervention group had significantly fewer medical service encounters (such as ambulance, police, or fire department), with a 49% decrease compared with the control group. The intervention group also had lower ED encounters (33% decrease), hospitalizations (14% decrease), and 30-day readmissions (57% decrease), although these differences were not statistically significant. Annualized costs per patient were about $3,500 lower in the intervention group, a significant decrease of approximately 20%. These savings were driven primarily by smaller inpatient costs for the intervention group, in particular by reducing the number of inpatient encounters among high utilizers (defined as those with multiple inpatient encounters during the study period).

Building better models

A healthy skepticism has tempered the initial exuberance for predictive models as organizations have struggled to integrate these models into clinical practice, or to measure meaningful improvement when they do. At the same time, several studies suggest that risk models may contribute to better health outcomes and reduced costs. Taken together, the mixed record indicates that using predictive analytics to support performance improvement is difficult, but worth pursuing.

To realize the promise of predictive analytics, organizations must follow best practices in the development and evaluation of risk models.

• The first question to consider is whether a new predictive model is necessary. There may be a preexisting risk score that adequately does the job, eliminating the need to develop a new model.
• Whether using a preexisting risk score or developing a new one, the score must be properly validated in the population of its intended use. Validation should assess both model discrimination (how well it sorts patients into different classes) and calibration (the correspondence between estimated and observed risk). The validation process should also include a simple and transparent baseline model, as that model may have performance commensurate with more complex predictive models.
• If the predictive model demonstrates improved accuracy over the baseline, as well as acceptable safety and fairness, then work may begin on the implementation. An essential component of the implementation is an impact analysis to determine whether the intervention is working and how it may be improved.
• Finally, it will greatly benefit the population health community if researchers publish both positive and null results because there is often just as much, if not more, to learn from interventions that fail to have the intended effect. A representative sample of impact analyses would be informative on the robustness of positive results, like those discussed above.

An important goal for every provider has always been to anticipate and treat emerging conditions before they become severe, and value-based care seeks to support and reinforce that effort. Even small reductions in acute care services, especially those involving inpatient encounters, can lead to substantial savings. More importantly, successful use of predictive models to target interventions where they are needed most can help fulfill healthcare’s core obligation of improving the quality of patient lives.

¹The algorithm was trained on a variety of features including pharmacy and utilization measures derived from claims data, lab results, and demographic information.

²Patients were eligible for the intervention if they were 65 years of age and older and were within the “middle” segment (the 6th to 50th percentile) of the cost pyramid in the year prior to their enrollment. This group was targeted because previous longitudinal research found that the middle segment was the costliest over a five-year period.