Established machine learning models to predict readmission for elderly patients with ischemic heart disease

Abstract

Background: The contribution of clinical features associated with 30-day or 1-year readmission in elderly patients with ischemic heart disease (IHD) and whether these features can be used to predict the readmission risk of patients has not been studied.
Aims: The study aimed to develop 30-day and 1-year readmission prediction models for elderly IHD patients using combined machine learning features routinely collected at the time of hospital discharge, and to investigate their prognostic impact.
Methods: Eight machine learning algorithms were used to develop prediction models. Area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) were used to assess discrimination. SHapley Additive exPlanations (SHAP) analysis was used to explain the contribution of features.
Results: A total of 6687 patients were enrolled. For 30-day readmissions, the categorical boosting (CB) model had the best predictive performance with the highest AUROC (0.72), and the Brier score was 0.23. For 1-year readmissions, the CB model had the best predictive performance with the highest AUROC (0.66), and the Brier score was 0.14. The age-adjusted Charlson comorbidity index, brain natriuretic peptide, heart failure, cholesterol, free thyroxine, thymidine kinase 1, osmotic pressure, and red blood cell distribution width (standard deviation) were the common important features to predict 30-day and 1-year readmissions of elderly IHD patients.
Conclusions: Elderly IHD patients with high risk of 30-day or 1-year readmission can be identified using machine learning and features collected at the time of discharge.

WHAT’S NEW? Few studies evaluated readmissions for elderly patients with ischemic heart disease. Therefore, we developed a predictive model based on machine learning. The results showed that the categorical boosting model had better predictive performance and good calibration in identifying readmissions in 30 days and 1 year for elderly patients with ischemic heart disease. We also found that the age-adjusted Charlson comorbidity index, brain natriuretic peptide level, heart failure, cholesterol, free thyroxine, thymidine kinase 1, osmotic pressure, and red blood cell distribution width (standard deviation) were common risk factors for readmission at different times.

INTRODUCTION

Ischemic heart disease (IHD), a common chronic non-communicable cardiovascular disease (CVD), is one of the major contributors to CVD-related disease burden [1]. Due to poor disease control, many IHD patients return to the hospital after discharge. For elderly IHD patients and their families, frequent readmissions may increase the financial burden and reduce the quality of life [2]. Therefore, identifying high-risk readmission patients and providing individualized therapeutic regi­mens is the key to reducing the readmission rate of elderly IHD patients.

Many features such as age [3], sex [4], length of stay (LOS) [5] , and complications [6] associated with patient readmission have been described. Results of some routine laboratory tests were also associated with rehospitalization [7, 8]. However, it has not been possible to determine the impact of these features on patient readmission rates or to assess the effect of these features on patient clinical outcomes. Moreover, it is not clear whether the elderly IHD patients at high risk of readmission can be accurately identified using these clinically relevant variables available at the time of hospital discharge. In recent years, some screening tools to identify heart failure or acute myocardial infarction patients who may be readmitted to the hospital have been established [9, 10], which showed that using machine learning to develop a complicated and reliable classification tool is possible. However, to our knowledge, there is no practical readmission evaluation tool for elderly IHD patients.

In this study, we produced 30-day and 1-year readmission prediction models for elderly IHD patients combining 8 machine-learning algorithms and data routinely collected at the time of hospital discharge. Additionally, we analyzed the contribution of those features to the predictions.

METHODS

Participants and outcome

The subjects were elderly IHD patients who were admitted to Sichuan Provincial People’s Hospital from August 2018 to April 2020. Inclusion criteria were age ≥60 years, and IHD diagnosis [11]. The exclusion criteria were as follows: 1) LOS <2 days; 2) transfer to other hospitals; 3) death in the hospital; and 4) follow-up time <1 year. The primary outcomes were 30-day and 1-year all-cause readmission. This study was approved by the Ethics Committee of the Sichuan Academy of Medical Sciences and Sichuan Provincial People’s Hospital (approval number: 2023-85). Due to the retrospective nature of the study, informed consent was waived.

Data collection and preprocessing

Data were collected from the Hospital Information System, which included basic patient information, medications, comorbidities, and laboratory tests. For multiple laboratory test results, we selected the last results before discharge. For multiple readmissions, the first admission record was included.

Data preprocessing was as follows. First, using information about the comorbidities at the time of discharge, we calculated the age-adjusted Charlson comorbidity index (ACCI) for all patients [12]. The calculation method of ACCI is shown in Supplementary material, Table S1. Moreover, we counted the number of medications (NOM) taken by each patient at the time of discharge. For the laboratory test results, the feature with more than 90% of missing data was deleted. Then, we used the random forest (RF) method to replace the missing data and Lasso analysis to select features. Before the modeling, the continuous variables were standardized by Z-score.

Model development

We used 8 representative machine learning algorithms including logistic regression (LR), k-nearest neighbor (KNN), support vector machine (SVM), adaptive boosting (AdaBoost), gradient boosting decision tree (GBDT), RF, categorical boosting (CB), and extreme gradient boosting (XGB) to develop prediction models.

In the modeling process, the patients were randomly assigned into a training set and a test set according to a proportion of 7:3. The training set was used to develop models, and 10-fold cross-validation on the training set was applied for parameter optimization. The test set was used to evaluate the predictive performance of the models. We developed the 30-day and 1-year readmission prediction models, respectively. Notably, compared to non-readmission, fewer patients were readmitted in 30 days. To improve the predictive performance of the models, we randomly matched the non-readmission patients to the 30-day readmission patients by a ratio of 5:1. The matched data were used to establish the 30-day readmission prediction model.

Model assessment

Model performance was mainly assessed using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Meanwhile, we calculated the accuracy, precision, positive predictive value (PPV), and negative predictive value (NPV) to evaluate the predictive performance of the machine learning models [13]. The calibration of the model was assessed by the Brier score, which is an index to evaluate both discrimination and calibration performance [14]. The model was considered to have favorable calibration when the Brier score was ≤0.25 [15]. The SHapley Additive exPlanations (SHAP) algorithm was used to measure the contribution of each feature to the best model.

Statistical analysis

All statistical analyses were performed using SPSS software version 25. Model building was implemented in Python (Version 3.7.0). The Shapiro–Wilk test was used to assess whether continuous variables followed normal distributions. Normally distributed variables and skewed distributional data were described as means and standard deviations (SD) and medians and interquartile ranges, respectively. The t-test or Mann–Whitney test was used to analyze significant differences, respectively. The χ2 test was chosen for the analysis of categorical variables, which were expressed as counts and percentages. A two-sided P-value of <0.05 was considered statistically significant.

RESULTS

Baseline characteristics

The patient data selection process is shown in Figure 1.

Figure 1. Patient data selection process

Abbreviations: IHD, ischemic heart disease; LOS, length of stay

This study included 6687 patients, of whom 904 (13.52%) experienced 1-year readmission and 325 (4.86%) experienced 30-day readmission. There were 3673 (54.93%) male patients and 3014 (45.07%) female patients. The average age was 73.5 (8.1) years. Data from all patients were used to develop machine learning models to predict 1-year readmission, and 1950 patients were used to predict 30-day readmission. Table 1 shows the general patient characteristics. The characteristics of 30-day readmission and non-readmission patients before matching are shown in Supplementary material, Table S2.

Table 1. Patient characteristics

Characteristics	30–day			1–year
	Readmission (n = 325)	Non-readmission (n = 1625)	P-value	Readmission (n = 904)	Non-readmission (n = 5783)	P-value
Sex
Males	185 (56.9)	863 (53.1)	0.208	557 (61.6)	3116 (53.9)	<0.001
Females	140 (43.1)	762 (46.9)		347 (38.4)	2667 (46.1)
Age, years
60–69	87 (26.8)	602 (37.0)	<0.01	267 (29.5)	2150 (37.2)	<0.001
70–79	143 (44.0)	648 (39.9)		371 (41.0)	2296 (39.7)
≥80	95 (29.2)	375 (23.1)		266 (29.4)	1337 (23.1)
ACCI
2–5	204 (62.8)	1198 (73.7)	<0.001	559 (61.8)	4214 (72.9)	<0.001
6–10	115 (35.4)	411 (25.3)		331 (36.6)	1508 (26.1)
≥11	6 (1.8)	16 (1.0)		14 (1.5)	61 (1.1)
NOM
1–5	66 (20.3)	335 (20.6)	<0.001	129 (14.3)	1182 (20.4)	<0.001
6–10	143 (44.0)	900 (55.4)		474 (52.4)	3233 (55.9)
≥11	116 (35.7)	390 (24.0)		301 (33.3)	1368 (23.7)
Total protein, g/l	66.8 (61.0–72.0)	67.8 (62.7–72.8)	<0.01	66.7 (61.0–71.8)	68.0 (62.8–72.7)	<0.001
Total bilirubin, µmol/l	13.3 (9.3–18.1)	13.3 (10.3–18.0)	0.333	12.8 (9.2–17.3)	13.4 (10.1–17.9)	<0.01
Cholesterol, mmol/l	3.78 (2.94–4.43)	4.07 (3.29–4.92)	<0.001	3.84 (3.13–4.63)	4.06 (3.32–4.88)	<0.001
AST, U/l	28 (22–37)	28 (22–36)	0.516	27 (22–35)	28 (22–36)	<0.05
Osmotic pressure, mOsm/l	283 (278–287)	283 (279–287)	<0.05	283 (278–287)	283 (279–287)	<0.05
Urea, mmol/l	6.75 (5.00–8.99)	6.13 (4.95–7.72)	<0.01	6.47 (4.95–8.53)	6.08 (4.86–7.78)	<0.001
Creatine kinase, U/l	83 (51–119)	92 (64–142)	<0.001	88 (59–129)	90 (63–138)	<0.05
Myoglobin, ng/ml	57.2 (39.2–100.8)	47.2 (33.8–75.0)	<0.001	55.2 (37.4–91.1)	48.0 (34.2–76.2)	<0.001
Creatinine, µmol/l	78.4 (64.1–103.6)	72.0 (59.9–89.8)	<0.001	78.7 (63.0–101.0)	72.2 (59.7–90.5)	<0.001
AST/ALT	1.30 (1.00–1.80)	1.28 (0.95–1.71)	0.167	1.28 (0.96–1.70)	1.28 (0.96–1.70)	0.511
eGFR, ml/min	79.0 (56.4–90.0)	84.1 (66.6–93.0)	<0.001	79.7 (57.7–90.5)	83.9 (65.3–93.1)	<0.001
HDL-C, mmol/l	1.14 (0.97–1.39)	1.18 (0.99–1.43)	0.084	1.16 (0.98–1.37)	1.19 (1.00–1.44)	<0.01
Triglyceride, mmol/l	1.24 (0.83–1.90)	1.40 (0.98–2.05)	<0.01	1.32 (0.91–1.96)	1.37 (0.97–1.99)	0.062
LDL-C, mmol/l	1.90 (1.36–2.61)	2.18 (1.58–2.86)	<0.001	1.98 (1.46–2.69)	2.18 (1.61–2.81)	<0.001
ALT, U/l	20 (15–30)	22 (16–34)	<0.01	21 (15–32)	22 (15–33)	0.098
GGT, U/l	28 (19–46)	26 (17–44)	0.088	27 (18–48)	25 (17–44)	<0.01
BNP, pg/ml	193.6 (58.6–511)	66.6 (26.2–214.1)	<0.001	112.5 (45.9–349.4)	72.7 (28.3–219.7)	<0.001
D–dimer, mg/l	0.65 (0.35–1.43)	0.46 (0.24–1.08)	<0.001	0.54 (0.28–1.19)	0.47 (0.25–1.13)	<0.01
Hemoglobin, g/l	124 (109–138)	130 (118–142)	<0.001	126 (112–139)	130 (117–141)	<0.001
Hematocrit	0.37 (0.31–0.42)	0.39 (0.33–0.42)	<0.01	0.38 (0.32–0.42)	0.39 (0.32–0.42)	<0.05
RDW–SD, fl	45.3 (43.0–48.1)	44.6 (42.5–47.0)	<0.01	45.1 (42.8–47.9)	44.7 (42.6–47.3)	<0.05
TK1, pmol/l	0.61 (0.30–1.31)	0.54 (0.23–1.24)	0.198	0.61 (0.33–1.32)	0.53 (0.23–1.23)	<0.01

Data presented as number (%) or median (IQR)

Abbreviations: ACCI, age-adjusted Charlson comorbidity index; ALT, alanine aminotransferase; AST, aspartate aminotransferase; BNP, brain natriuretic peptide; eGFR, estimated glomerular filtration rate; GGT, γ-glutamyl transpeptidase; HDL-C, high density lipoprotein cholesterol; LDL-C, low density lipoprotein cholesterol; NOM, number of medications; RDW-SD, red blood cell distribution width-standard deviation; TK1, thymidine kinase 1

Features selection

A total of 174 features were collected, and 85 features with missing data >90% were deleted (Supplementary material, Table S3). Then, Lasso regression analysis was performed on the remaining 89 features as independent variables with 30-day and 1-year readmissions as dependent variables, respectively (Figure 2). When using Lasso regression for features selection, the appropriate lambda values can be select the lambda with minimum mean square error (dashed black line in Figure 2), in which case the feature with a non-0 coefficient is the important feature to be selected. The names of selected features are presented in Supplementary material, Table S4.

Figure 2. A. and B. Lasso regression analysis for 30-day readmission. C. and D. Lasso regression analysis for 1-year readmission. A. and C. Vertical lines are drawn at selected values by applying 10-fold cross-validation. B. and D. In the Lasso model, the coefficient profiles of 89 texture features were draw from the log (λ) sequence. Vertical dotted lines are drawn at the minimum mean square error and the standard error of the minimum distance

Model performance and calibration

Eight machine learning algorithms combined with 10 features selected by Lasso were used to develop prediction models for 30-day readmission. On the training set, the RF model had the highest AUROC (0.95 [0.01], standard error = 0.003) (Figure 3A). On the test set, the CB model had the highest AUROC (0.72) (Figure 3B). The GBDT and XGB models had the highest AUPRC (0.31) on the test set (Figure 3C). The AUPRC of the CB model was 0.30 on the test set (Figure 3C). The accuracy, precision, PPV, and NPV of the CB model to 30-day readmission on the test set were 0.77, 0.28, 0.28, and 0.87 (Table 2), respectively.

Figure 3. Models predictive performance. A., B. and C. 30-day readmission. A. The AUROC on the training set. B. The AUROC on the test set. C. The AUPRC on the test set. D., E. and F. 1-year readmission. D. The AUROC on the training set. E. The AUROC on the test set. F The AUPRC on the test set

Abbreviations: AUPRC, area under the precision recall curve; AUROC, area under the receiver operating characteristic curve; other — see Table 2

Table 2. Models performance

Prediction	Model	Accuracy	Precision	PPV	NPV	Brier score
30-day readmission	LR	0.63	0.26	0.26	0.92	0.37
	KNN	0.57	0.23	0.23	0.90	0.43
	SVM	0.59	0.25	0.25	0.93	0.41
	AdaBoost	0.71	0.28	0.28	0.89	0.29
	GBDT	0.68	0.26	0.26	0.89	0.31
	RF	0.78	0.30	0.30	0.87	0.22
	CB	0.77	0.28	0.28	0.87	0.23
	XGB	0.81	0.37	0.37	0.88	0.19
1-year readmission	LR	0.62	0.18	0.18	0.91	0.38
	KNN	0.41	0.15	0.15	0.93	0.59
	SVM	0.69	0.19	0.19	0.91	0.31
	AdaBoost	0.83	0.22	0.22	0.88	0.17
	GBDT	0.87	0.21	0.21	0.88	0.13
	RF	0.87	0.30	0.30	0.88	0.13
	CB	0.86	0.17	0.17	0.88	0.14
	XGB	0.86	0.26	0.26	0.88	0.14

Abbreviations: AdaBoost, adaptive boosting; CB, categorical boosting; GBDT, gradient boosting decision tree; KNN, k-nearest neighbor; LR, logistic regression; NPV, negative predictive value; PPV, positive predictive value; RF, random forest; SVM, support vector machine; XGB, extreme gradient boosting

Eight machine learning algorithms combined with 36 features selected by Lasso were used to develop prediction models for 1-year readmission. On the training set, the RF model had the highest AUROC (0.98 [0.01], standard error = 0.002) (Figure 3D). On the test set, the CB model had the highest AUROC (0.66) (Figure 3E). Moreover, LR, CB, and XGB models had the highest AUPRC (0.20) on the test set (Figure 3F). The accuracy, precision, PPV, and NPV of the CB model for 30-day readmission on the test set were 0.86, 0.17, 0.17, and 0.88 (Table 2), respectively.

We used the Brier score to assess the calibration of the machine learning models. For 30-day readmission, the XGB model had the optimal Brier score (0.19). The Brier score of the CB model was 0.23 (Table 2). For 1-year readmission, the Brier score of the CB model was 0.14 (Table 2).

Features contribution

The contributions of features to prediction of best-performing models by different periods of readmission are presented in Figure 4. For 30-day readmission, the most important features were ACCI, B-type natriuretic peptide (BNP), heart failure, cholesterol, and thymidine kinase 1 (TK1) (Figure 4A). For 1-year readmission, the most important features were sex, hypertension, nephropathy, NOM, and BNP (Figure 4B).

Figure 4. Contributions of input features to readmission predictions. The SHAP summary plot of the 10 most important variables of the CB model for 30-day readmission (A) and 1-year readmission (B). In the plots, the x-axis indicates the SHAP value; and the y-axis indicates the relationship between features and SHAP values. The color of the dot represents the value of the features. Red represents higher feature values, and blue represents lower feature values (C). D. Scatter plot shows the relationship between the feature value and SHAP value. C. BNP for 30-day readmission. D. NOM for 1-year readmission. In the plots, the distributions of the SHAP value and feature values are shown as histo­grams on the right and top of the scatter graph. E. and F. Contribution of every feature to the predicted outcome of one sample. Red represents positive contribution, and blue represents negative contribution. E. 30-day readmission. F. 1-year readmission

Abbreviations: ADA, adenosine deaminase; FT4, free thyroxine; HS-TNTI, high-sensitivity cardiac troponin I; OSM, osmotic pressure; SHAP, Shapley additive explanations; TT3, total triiodothyronine; other — see Table 1 and Figure 1

The relationship between SHAP values and feature values is illustrated in more detail for the features of patient BNP and NOM in Figure 4C (30-day readmission) and Figure 4D (1-year readmission). The results showed that patients with high BNP may have a high risk of 30-day readmission. Similarly, patients with ≥8 NOM were more likely to be readmitted in 1 year after discharge.

Furthermore, SHAP can contribute to every feature of the predicted outcome of each patient. An example of 30-day readmission is illustrated in Figure 4E. Total triiodothyronine (TT3), BNP, osmotic pressure (OSM), free thyroxine (FT4), TK1, and red blood cell distribution width standard deviation (RDW-SD) provided a positive contribution, while heart failure, ACCI, and cholesterol provided a negative contribution. Another example of 1-year readmission is illustrated in Figure 4F. High-sensitivity cardiac troponin I, LOS, total bilirubin, TK1, and procalcitonin provided a positive contribution, while sex, nephropathy, hypertension, and heart failure provided a negative contribution.

DISCUSSION

In this study, we reported the development and validation of machine learning models to identify elderly IHD patients at high risk for 30-day or 1-year readmission. We used 8 representative machine learning algorithms including LR, KNN, SVM, AdaBoost, GBDT, RF, CB, and XGB, combined with the features selected by Lasso analysis to develop prediction models. For 30-day or 1-year readmission, the CB model had the best predictive performance.

Several studies have assessed 30-day or 1-year all-cause readmissions after acute myocardial infarction, but few are specific to IHD, especially in elderly patients. Dodson et al. [16] established a risk model by backward and Bayesian selection to predict all-cause readmission at 30 days for 3006 elderly acute myocardial infarction patients (≥75 years). The C statistic of the model was 0.63 in the validation cohort. Another study established RF was the best predictive model with an AUROC of 0.66 [17]. The model developed by Dreyer et al. [18] to predict 1-year readmission among younger acute myocardial infarction adults had good calibration, and the C statistic of the model was 0.69. Compared to other studies, the best model in predicting 30-day or 1-year readmissions in our study had a consistent predictive performance. Meanwhile, the Brier score of the CB model was 0.23 on 30-day readmission and 0.14 on 1-year readmission, which showed the model had a good calibration ability.

An important challenge in optimizing the clinical utility of machine learning models resides in balancing the PPV and NPV in specific clinical settings [13]. As the proportion of patients at high risk of readmission is low, benefit from machine learning models should ensure that patients with high risk of readmission can be identified before discharge. In our study, all machine learning models had good NPV, which suggested that our model can accurately identify patients at low risk for readmissions. However, the PPV values of all models were unsatisfactory, and we need to optimize these models to improve the identification of high-risk readmission patients in further studies.

Many of the top predictors contributing to identified readmission of elderly IHD patients in this study have been previously reported, such as BNP, LOS, sex, and ACCI [19–23]. Heart failure is one of significant risk factors for 30-day readmission in cardiovascular patients [24]. Our results also clarify that elderly IHD patients with heart failure were more likely to be readmitted within 30 days or 1 year. Poly-comorbidity often means polypharmacy. Care for hospitalized elderly patients is made more complex by polypharmacy, which may increase the risks of adverse drug events [25]. Our study was unable to trace the cause of readmission. This is one of the limitations of the study. However, those elderly IHD patients who are affected by polypharmacy should receive more attention to reduce readmissions. Furthermore, a new finding is that OSM is also a common important feature for the prediction of 30-day or 1-year readmission of elderly IHD patients. Severe reduction in OSM predicts adverse outcomes in patients with cardiopulmonary failure. Taniguchi and colleagues recommended using OSM measurement to monitor patients with unstable angina pectoris [26]. One interesting finding was that TK1 was a common important feature in predicting 30-day and 1-year readmission in elderly IHD patients. However, TK1 is not routinely tested in patients with cardiovascular disease, which limits the clinical utility of the model.

Although our results provided overall risk estimates for the population, they are of limited clinical assistance because significant individual differences existed in readmission risk features. The same features may have different clinical significance for different patients. A major advantage of the model developed in our study is that it provides individual risk estimates, which may assist clinicians in providing optimal care and appropriate interventions for patients.

Limitations

There are several limitations to this study. First, although we had information about the complications in the elderly IHD patients who we had enrolled, some relevant aspects of that information, such as the duration of the complications and disease severity were not included in our study. Second, this was a single-center study, and it would need to be tested in other medical institutions to assess the predictive performance of the model. Third, the inclusion of some unconventional laboratory tests such as TK1 in the final model may limit its clinical utility. Finally, the inclusion of additional features, such as health literacy, frailty, and socioeconomic status, could improve the readmission risk assessment and should be validated in further studies.

CONCLUSIONS

In this study, we used clinically relevant features available at the time of hospital discharge combined with 8 machine learning algorithms to predict 30-day and 1-year readmission in elderly IHD patients and investigated features contributing to these predictions. The CB model had the best predictive performance and good calibration, which showed that machine learning models had potential clinical application value.

Supplementary material

Supplementary material is available at https://journals.viamedica.pl/polish_heart_journal.

Article information

Conflict of interest: None declared.

Funding: This research was funded by the National Key Research and Development Program of China (2020YFC2005506).

Open access: This article is available in open access under Creative Common Attribution-Non-Commercial-No Derivatives 4.0 International (CC BY-NC-ND 4.0) license, which allows downloading and sharing articles with others as long as they credit the authors and the publisher, but without permission to change them in any way or use them commercially. For commercial use, please contact the journal office at polishheartjournal@ptkardio.pl