蘑菇视频

Hospitals can be dangerous places for patients. While in the hospital, patients run the risk of contracting such as sepsis and Clostridioides difficile or experiencing other adverse events such as falls. Hospitals have limited resources, requiring efficient use of beds and clinician time. For these reasons and more, it is in the best interest of patients, hospitals, and public health to limit any hospital stay to no longer than necessary.听

Recognizing the need to support shorter lengths of stay, Dr. John Fahrenbach, a data scientist at the University of Chicago Medicine (UCM), developed a machine learning model that used clinical characteristics to identify patients most suitable for discharge after 48 hours. Using this tool, the hospital could ensure the timely release of specific patients by allocating and prioritizing care management resources, including discharge planning, home health services, and clinician or patient administrative assistance.

During the development process, Dr. Fahrenbach鈥檚 team determined that including zip codes as a feature increased the model鈥檚 predictive accuracy. After introducing zip codes into the model, however, a team member who reviewed the output raised concerns. 鈥淲e know Chicago鈥檚 patient population and knew something was off when stratifying the model by race,鈥� said Dr. Fahrenbach.

The zip codes in UCM鈥檚 service area reflect the . Trends in the data showed patients from predominantly Caucasian and relatively affluent zip codes were more likely to have shorter lengths of stay than others. Conversely, the zip codes correlating to longer lengths of stay were predominantly African American and limited income. Had the model been deployed, it would have resulted in prioritizing care management services to a population with less need for those services, diverting them from patients more at risk and in greater need of enhanced care management.听

To confirm the model perpetuated historical biases in the UCM patient population, the team tested it for racial bias by stratifying the results by race. This test of the model confirmed the concerns, and the team halted the development process.

Dr. Fahrenbach鈥檚 team met with UCM鈥檚 diversity, inclusion, and equity department to develop approaches to avoid such biases in the future. Within UCM鈥檚 diversity department, the team collaborated with Dr. Marshall Chin, physician, professor of healthcare ethics, and expert in social determinants of health. Dr. Chin observed that the data analytics team 鈥渉ad good intentions to improve efficiency and patient safety, but problems arose because they initially did not have an equity lens when developing the algorithm.鈥�

According to Dr. Chin, healthcare AI needs to apply 鈥渁n equity lens at all stages of development and deployment to avoid negative unintended consequences鈥攁nd better yet, have a positive effect on health equity.鈥� The diversity department and the data science team worked together to create 鈥渆quity checkpoints鈥� to identify sources of bias and unfairness in future AI applications at each stage of development: conceptualization, development, deployment, and monitoring. At each equity checkpoint, the team asks themselves if they are improving the health outcomes of all patients or unfairly treating any at-risk patient group.

With no off-the-shelf products for facilitating equity checkpoints in healthcare-focused AI applications, Dr. Chin and Dr. Fahrenbach developed their own equity checkpoint system using a few general principles:听

• Examine the algorithm with bias in mind
  Audit your algorithm for potential sources of bias and outputs that contribute to unequal outcomes among patient populations.
  
  
• Look for flaws in your data that could lead to bias
  Evaluate your data in search of flaws that could result in biased algorithmic outputs. For example, is the available data for socially at-risk patient populations less accurate or complete than the data available for more advantaged populations?
  
  
• Take steps to ensure fairness
  Consider how to ensure fairness across different patient populations when the models are developed and deployed. This could include adjusting thresholds within the model to ensure equal outcomes and allocation of healthcare resources.
  
  
• Use continuous monitoring after the algorithm is deployed
  After deployment, developers and users should continue to monitor data for inequalities that may have been missed during development or can only be detected in a real practice setting.
  
  
• Listen to providers and patients
  Across all stages of development and deployment, developers should conduct discussions with both providers and patients. The voice of patients鈥攖he end users of clinical care鈥攊s especially important.听

鈥淭he AI field is fluid and evolving quickly. You need to run equity checks on off-the-shelf AI algorithms since you usually can鈥檛 modify these AI algorithms. If major equity issues are found, this should be addressed immediately before deployment,鈥� said Dr. Fahrenbach. This case demonstrates the importance of safeguards in AI development to prevent the deployment of an algorithm with potentially unfair and even harmful effects on patients.

This case also illustrates just how crucial it is to account for factors such as social risk and other social determinants of health when developing healthcare AI. Appropriate representation of socially at-risk patient populations in machine learning datasets is essential to ensuring these tools serve to improve health equity.

As an example, the National Institutes of Health鈥檚 initiative aims to collect genetic and health data from one million Americans鈥攑articularly those from socially at-risk groups that have been historically underrepresented in medical studies and datasets. By 2020, half of the participants in the program were racial and ethnic minorities, potentially expanding knowledge about people who have historically been overlooked in clinical research.

With data such as those collected from the All of Us initiative, researchers can identify and address occurrences of underdiagnosis in smaller health system datasets by comparing them to these types of broadly representative datasets. To advance health equity and access for patients from marginalized communities, datasets must enlist criteria for determining how patients in these communities differ from the population overall.

Finally, the UCM experience demonstrates that, to address algorithmic bias, we must cultivate a strong feeling of psychological safety within development teams. is a concept rooted in organizational theory that seeks to empower team members to feel safe bringing up ideas or concerns that may differ from the thinking of the overall team. The issue with including zip codes as a variable in the length of stay algorithm was initially raised by a member of the development team. Without UCM鈥檚 culture of psychological safety, this team member may not have felt empowered to speak up and raise a potentially harmful issue with the model outputs.

UCM鈥檚 success with identifying and addressing bias in its length of stay algorithm demonstrates it is possible to put in place protocols that ensure the needs of socially at-risk patient populations are addressed when designing, deploying, and evaluating AI algorithms in healthcare. Ultimately, the length of stay algorithm was not deployed. But it did provide a useful learning experience for implementing processes that will enable the UCM team to identify sources of bias when developing healthcare AI models in the future.

Want to learn more about bias in healthcare AI? Read our next blog, How to Mitigate Bias in Healthcare Algorithms.

听