[MA 2024 12] Bridging the gap between Machine Learning research and clinical practice: testing Machine Learning models for hospital readmission prediction in a Dutch mid-sized hospital.
Researchgroup Predicaid, Amsterdam/Rotterdam - Spaarne Gasthuis Hospital, Haarlem - Department of Medical Informatics, Amsterdam UMC, university of Amsterdam
Proposed by: K. Steur [karel@predicaid.nl]
Introduction
Unexpected hospital readmissions within 30 days following an initial admission represent a significant issue in hospitals. These readmissions are prevalent, costly, and exacerbate the burden on hospital capacity. In the Netherlands, the rate of 30-day readmissions varies, ranging from 6% to 14% across different hospitals, with a national average of 9.8% in 2022. These readmissions resulted in an additional 103,342 emergency room consultations and admissions to inpatient departments, accounting for an estimated 550,000 extra patient days. This exacerbates the strain on hospital capacity, which is already under pressure due to staff shortages and increased morbidity associated with an aging population. The financial impact on the Dutch healthcare system is substantial, with estimated costs ranging from 150 to 400 million euros annually. Furthermore, each readmission is associated with poorer health outcomes compared to patients who are not readmitted. Effectively reducing readmissions could benefit patients and hospitals, including physicians, nurses, and patient departments.
Research indicates that 30% to 50% of all readmissions are potentially preventable. Although efforts to identify patients at high risk of readmission have led to the development of predictive scores based on clinical variables, these models demonstrate only moderate effectiveness in predicting readmissions. Currently, no prediction model utilizing advanced technologies, such as machine learning, has been validated in the Netherlands for predicting readmissions.
Machine learning-based prediction models are gradually being integrated into patient care in hospitals. Extensive and validated models for the prediction of hospital readmissions are already in use in the USA, indicating a promising opportunity for similar applications in Dutch hospitals. Moreover, in the Netherlands, machine learning models are already being utilized in practical applications or undergoing advanced validation for various other use cases, such as predicting ICU discharge/mortality and assessing perioperative infection risk.
Our research group is collaborating with a mid-sized Dutch hospital, the Spaarne Gasthuis, to conduct a six-month research pilot. This pilot comprises two simultaneous processes. 1) Model Training: We will refine our pre-trained model using two different anonymized datasets. This involves incorporating structured data from general departments such as geriatrics, surgery, and internal medicine to enhance the model's accuracy and applicability. 2) End-User Analysis: Through a combination of interviews, workflow mapping, literature review, and prototype testing, we aim to comprehensively understand the patient discharge and readmission processes. This approach will help us identify key factors and potential areas for intervention.
Description of the SRP Project/Problem
In this SRP, you will help in bridging the gap between machine learning research and clinical practice. Collaborating with data scientists, you will engage in the development and validation of machine learning prediction models, including gradient boosting models. Your responsibilities will include training and validating a prediction model using two distinct new datasets, leveraging techniques that have been validated in other countries.
Furthermore, you will work closely with a physician to investigate the discharge process comprehensively. This will involve frequent interactions with other physicians and nurses in the hospital to gain an in-depth understanding of the discharge and readmission processes. The objective is to identify potential areas where a prediction model could provide significant value. We will employ hierarchical cluster analysis and the card sorting method to analyze the themes identified from interviews with physicians and nurses.
Research questions
Can we collaboratively develop a prediction model that assists physicians and nurses in optimizing discharge decisions and plans, leading to a clinically significant reduction in hospital readmissions?
Model Training:
- What clinical variables are critical for predicting readmissions
- What is the minimum and optimal sample size required to achieve a sufficient Area Under the Curve (AUC)?
- Which model provides the best AUC?
End-User Analysis:
- What is the detailed structure of the discharge process, and where are the bottlenecks?
- What are the primary factors contributing to readmissions, and what potential technical or non-technical solutions can address these factors?
- What additional information provided by a prediction model is beneficial, and at what point in the discharge process do healthcare providers prefer to receive this information?
Expected results
The main outcomes of this project will be 1) a first validation of machine learning models for the prediction of hospital readmissions 2) a scientific paper to be submitted to a top-tier journal or conference. The scientific paper will be the students master’s thesis.
Time period
November – June
May – November
Contact
Karel Steur, MD
karel@predicaid.nl
References
Jiang LY et al. Health system-scale language models are all-purpose prediction engines. Nature. 2023 Jul;619(7969):357-362. doi: 10.1038/s41586-023-06160-y. Epub 2023 Jun 7. PMID: 37286606; PMCID: PMC10338337.
Grossman Liu L et al. Published models that predict hospital readmission: a critical appraisal. BMJ Open. 2021 Aug 3;11(8):e044964. doi: 10.1136/bmjopen-2020-044964. PMID: 34344671; PMCID: PMC8336235.
Mahmoudi E et al. Use of electronic medical records in development and validation of risk prediction models of hospital readmission: systematic review. BMJ. 2020 Apr 8;369:m958. doi: 10.1136/bmj.m958. PMID: 32269037; PMCID: PMC7249246.
Thoral PJ et al. Explainable Machine Learning on AmsterdamUMCdb for ICU Discharge Decision Support: Uniting Intensivists and Data Scientists. Crit Care Explor. 2021 Sep 10;3(9):e0529. doi: 10.1097/CCE.0000000000000529. PMID: 34589713; PMCID: PMC8437217.
Tricia Baird et al. Reducing Readmission Risk Through Whole-Person Design. 2023 NEJM Catal Innov Care Deliv 2023;4(1) DOI: 10.1056/CAT.22.0237 VOL. 4 NO.
van der Meijden SL et al. Identifying and Predicting Postoperative Infections Based on Readily Available Electronic Health Record Data. Stud Health Technol Inform. 2023 May 18;302:348-349. doi: 10.3233/SHTI230134. PMID: 37203678.
Maurer PP, Ballmer PE. Hospital readmissions--are they predictable and avoidable? Swiss Med Wkly. 2004 Oct 16;134(41-42):606-11. doi: 10.4414/smw.2004.10706. PMID: 15592954.
Fluitman KS et al. Exploring the preventable causes of unplanned readmissions using root cause analysis: Coordination of care is the weakest link. Eur J Intern Med. 2016 May;30:18-24. doi: 10.1016/j.ejim.2015.12.021. Epub 2016 Jan 13. PMID: 26775179.
Hekkert K et al. Re-admission patterns in England and the Netherlands: a comparison based on administrative data of all hospitals. Eur J Public Health. 2019 Apr 1;29(2):202-207. doi: 10.1093/eurpub/cky199. PMID: 30445564.
Hwang AB et al. External validation of EPIC's Risk of Unplanned Readmission model, the LACE+ index and SQLape as predictors of unplanned hospital readmissions: A monocentric, retrospective, diagnostic cohort study in Switzerland. PLoS One. 2021 Nov 12;16(11):e0258338. doi: 10.1371/journal.pone.0258338. PMID: 34767558; PMCID: PMC8589185.
Kwong JCC et al. Integrating artificial intelligence into healthcare systems: more than just the algorithm. NPJ Digit Med. 2024 Mar 1;7(1):52. doi: 10.1038/s41746-024-01066-z. PMID: 38429418; PMCID: PMC10907626.
Mohanty SD et al. Machine learning for predicting readmission risk among the frail: Explainable AI for healthcare. Patterns (N Y). 2021 Dec 3;3(1):100395. doi: 10.1016/j.patter.2021.100395. PMID: 35079714; PMCID: PMC8767300.
Singotani RG et al. Towards a patient journey perspective on causes of unplanned readmissions using a classification framework: results of a systematic review with narrative synthesis. BMC Med Res Methodol. 2019 Oct 4;19(1):189. doi: 10.1186/s12874-019-0822-9. Erratum in: BMC Med Res Methodol. 2019 Nov 27;19(1):214. doi: 10.1186/s12874-019-0851-4. PMID: 31585528; PMCID: PMC6778387.
Goldstein BA et al. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J Am Med Inform Assoc. 2017 Jan;24(1):198-208. doi: 10.1093/jamia/ocw042. Epub 2016 May 17. PMID: 27189013; PMCID: PMC5201180.
Balagopalan A et al. Machine learning for healthcare that matters: Reorienting from technical novelty to equitable impact. PLOS Digit Health. 2024 Apr 15;3(4):e0000474. doi: 10.1371/journal.pdig.0000474. PMID: 38620047; PMCID: PMC11018283.
Blunt I et al. Classifying emergency 30-day readmissions in England using routine hospital data 2004-2010: what is the scope for reduction? Emerg Med J. 2015 Jan;32(1):44-50. doi: 10.1136/emermed-2013-202531. Epub 2014 Mar 25. PMID: 24668396; PMCID: PMC4283684.
Van Grootven B et al. Prediction models for hospital readmissions in patients with heart disease: a systematic review and meta-analysis. BMJ Open. 2021 Aug 17;11(8):e047576. doi: 10.1136/bmjopen-2020-047576. PMID: 34404703; PMCID: PMC8372817.
Goodman DM et al. Development and Validation of an Integrated Suite of Prediction Models for All-Cause 30-Day Readmissions of Children and Adolescents Aged 0 to 18 Years. JAMA Netw Open. 2022 Nov 1;5(11):e2241513. doi: 10.1001/jamanetworkopen.2022.41513. PMID: 36367725; PMCID: PMC9652755.
Andaur Navarro CL et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021 Oct 20;375:n2281. doi: 10.1136/bmj.n2281. PMID: 34670780; PMCID: PMC8527348.