Study on medical dispute prediction model and its clinical-application effectiveness based on machine learning

Abstract

Background

Medical dispute is a global public health issue, which has been garnering increasing attention. In this study, we used machine learning (ML) method to establish a dispute prediction model and explored the clinical-application efficiency of this model in effectively reducing the occurrence of medical disputes.

Methods

Retrospective study of All disputes filed by Gansu Medical Mediation Committee from 2019 to 2021 and patients with the same hospital level as that of the dispute group and hospitalization year were randomly selected as the control group in 1:1 ratio. SPSS software was used for univariate feature selection of the 14 factors that may cause disputes, and factors with statistical differences were selected. The data were divided into training and test sets in a 7:3 ratio. Six ML models were selected, and Python was used to establish a dispute prediction model. The area under the curve (AUC) of the receiver operating characteristic curve (ROC), sensitivity, specificity, accuracy, precision, average precision (AP), and F1 score were used to characterize the fitting and accuracy of the models, while decision curve analysis (DCA) was used to evaluate their clinical utility.

Results

A total of 1189 patients in the dispute and control groups were extracted. Following 11 influencing factors were selected: the inpatient department, doctor title, patient age, patient gender, patient occupation, payment method, hospitalization days, hospitalization times, discharge method, blood transfusion volume, and hospitalization espenses. Compared to other models, the AUC (0.945, 95% CI 0.913–0.981), Sensitivity (0.887), Accuracy (0.887), AP (0.834), and F1 score (0.880) of the random forest model were higher than those of other models, while the DCA curve indicated its high clinical benefits.

Conclusions

Inpatient department, hospitalization expenses, and discharge type are the primary influencing factors of dispute. Random forest exhibited high dispute prediction and clinical-application value and is expected to be promoted for offline dispute prediction.

Introduction

Medical dispute typically refers to a scenario in which doctors and patients have different understandings of the medical consequences and their causes. Thus, the patients propose responsibility or compensation to the health administrative department or judicial organ [1]. Statistically, 42.2% of physicians in the United States have been sued in their careers, and 22.4% of them have been sued twice or more [2]. The United States spends approximately 55 billion dollars annually to tackle medical disputes [3]. Studies have reported that 33.48–76% of medical workers have experienced medical disputes or workplace violence worldwide [4, 5]. According to Fan Z et al., after the occurrence of medical disputes, the involved medical personnel may suffer from anxiety and depression, which discourage their work enthusiasm and hinder the development and exploration of medical diagnosis and treatment technology [6]. In October 2018, the state council promulgated the Regulations on the Prevention and Handling of Medical Disputes. The prevention of medical disputes was included in the regulations for the first time. Therefore, research on operable mathematical models for the prediction of clinical medical disputes has attracted increasing attention [7].

Currently, machine learning (ML) is widely used in the medical field [8, 9]. As a branch of artificial intelligence (AI), it can overcome the shortcomings of traditional logistic regression and mathematical models and has a strong capability for feature recognition, classification, and prediction [10]. ML techniques provide more accurate diagnostic techniques and personalized patient therapy, They have played an increasingly important role in classification and prediction problems, Previous studies have shown that ML provides relatively accurate results in the classification of epidemiological data [11,12,13,14].

A medical dispute prediction model that can accurately evaluate risk probability of medical dispute can help in personalized interventions. Yi et al. established a prediction model of medical disputes between health care workers and patients in terms of hospital Legal construction based on machine learning technology [15], and Ou et al. modeled medical risk warning based on Hospital Information System(HIS) data mining [16]. All the above studies are for the establishment of the medical dispute prediction model, without discussing the clinical efficacy of the model, nor specifically focusing on the dispute prediction of inpatients. Besides, the dispute data is small or there is no dispute data, so the model is not convincing. The main reasons are: most medical institutions or clinical departments are afraid regarding hampering their own reputation or privacy issues. Thus, they tackle disputes privately or conceal them and do not prefer to report them to the management. Hospital management and departments do not record information on medical dispute complaints, and most dispute data are not saved or incomplete. Therefore, establishing a scientific medical dispute prediction model and effectively applying it to clinical practice is challenging [17].

In this study, we extracted 2,331 complete inpatients medical dispute data from Gansu Medical Mediation Committee, used various ML methods, and A model with high dispute prediction ability is obtained, and the model has good clinical application efficiency. Therefore, this study could identify medical disputes in advance, thus aiding early provision of medical dispute interventions, reduce the occurrence of medical disputes in inpatients and carry out clinical work smoothly.

Materials and methods

Data selection

A total of 2,331 medical dispute cases filed by the Gansu Medical Mediation Committee from January 2019 to December 2021 were extracted. The control group was randomly selected in 1:1 ratio. Following specific operation process was followed. First, the data of all patients discharged from Gansu Provincial Health and Health Commission in the same month as the patient in dispute, with the same hospital level and no disputes, were extracted. Then, the random sampling tool of Excel was used for sampling until complete and available control data were extracted. The key factors that may cause disputes in the dispute and control groups were extracted, including 14 factors: inpatient department, dispute quarter, doctor title, patient age, patient gender, patient occupation, patient marital status, patient nationality, payment method, hospitalization days, hospitalization times, discharge type, blood transfusion volume, and hospitalization expenses. The Ethics Review Committee of the Second Hospital of Lanzhou University approved the research protocol and exemption of informed consent [2021 A-262].

Data processing

Quality control of the samples: The dispute data consisted of all inpatients from public hospitals in Gansu province who complained to the Gansu Medical Mediation Committee regarding their doubts about the diagnosis and treatment methods, results, and attitudes towards medical care. All cases included the patients who had been discharged from the hospital. All private hospital data, outpatient, and emergency patients were excluded; patient information with some vacant indicators or recording errors was excluded.

In addition to the patient age, hospitalization days, hospitalization times, blood transfusion volume, and hospitalization expenses of the patient as continuous variables, nine other categorical variables were classified based on data type and clinical experience, as presented in Appendix A.

Model building

Datasets and algorithms

The 14 factors influencing medical disputes were analyzed retrospectively, and those with statistical differences were placed in the final model operation. Logistic regression, Random forest, Decision Tree classifier, Gaussian naive bayes(NB), Bagging classifier, and AdaBoost classifier were used to establish the ML model owing to their good performance in clinical classification prediction [18]. Python 3.7.11 was used for model construction and statistical analysis. Stratification was performed based on the outcome, and the dataset was randomly divided into training and test sets in a 7:3 ratio.

Model evaluation

The abovementioned six models were trained separately and, then, tested using the test set data. The models were verified by 10-fold cross-validation method. The area under the curve (AUC) of the receiver operating characteristic curve (ROC), Best cutoff (the maximum value of the Yunden index), sensitivity, specificity, accuracy, positive predictive value (PPV), average precision (AP), and F1 score were used to characterize the fitting and accuracy of the model [7]. The higher the value of each index, the better the model performance. The population stability index (PSI) was used to measure the stability of the model in the training and test sets [19]. The model was considered stable, slightly unstable, and unstable for PSI < 0.1, 0.1 < PSI < 0.25, and PSI > 0.25, respectively. The goodness of fit is determined from the coefficient of determination R² with the maximum value of 1. The closer the value of R² is to 1, the better the regression line fits the observed value. Decision Curve Analysis (DCA) was used to evaluate the clinical benefit of these models [20], The larger the area under the curve, the more the model benefits the patient [20]. Moreover, coefficients of weight importance in the final model were provided to rank the importance of the feature.

The 95% confidence interval for the AUC is calculated as follows: lower_bound = AUC_mean-1.96*AUC_se, upper_bound = AUC_mean + 1.96*AUC_se [AUC_mean = AUC/π, AUC_se = sqrt((1-AUC^2)/n)], where n is the sample size. In this study, the confidence interval can be calculated automatically by using Python software.

Statistical analysis methods

Continuous variables were expressed as mean ± standard deviation (SD), and statistical differences were estimated using the independent sample T-test or Mann–Whitney U test. Categorical variables were expressed as frequency (percentage), and the differences between groups were compared using the Chi-square or Fisher exact test. Hosmer–Lemeshow test was used to assess the calibration of models. Bilateral P < 0.05 was considered statistically significant. IBM SPSS 23 software was used to perform the above analyses. Python 3.7.11 was used for model calculation and various icon making.

Result

Results of univariate analysis

The 2,331 dispute-involved patients were screened and 1,189 patients were selected that met the inclusion and exclusion criteria. Thus, a total of 2,378 patients in the dispute and control groups were finally determined according to a 1:1 ratio. The data were divided into a training set (1903) and a test set (475) in a 7:3 ratio.The above results are detailed in Fig. 1.

Fig. 1

Data processing flow chart

According to the statistical tests of 14 suspected influencing factors, no significant differences were observed in the dispute quarter, patient marital status, and patient nationality (P > 0.05). There were differences in inpatient department, doctor title, patient age, patient gender, patient occupation, discharge type, hospitalization days, hospitalization times, blood transfusion, hospitalization expenses and discharge type (P < 0.05), as presented in Tables 1 and 2.

Table 1 Chi-square test of ten classified variables

Table 2 Z-test results of four continuous variables

Evaluation of the model

Model training and testing were conducted on 11 variables influencing medical disputes with statistical differences. ROC and Precision Recall(PR) curves of the test set of six models are shown in Fig. 2. The area under random forest curve was the largest (AUC = 0.945), followed by AdaBoost classifier (AUC = 0.925) and bagging classifier (AUC = 0.924). Gaussian NB (AUC = 0.846), decision tree classifier (AUC = 0.838), and logistic regression (AUC = 0.826) were consistent with the PR curve. Detailed performance parameters of each model are listed in Tables 3 and 4. Stable between all model training and test sets (PSI < 0.1). In addition to the Specificity and PPV (0.886/0.879), the accuracy (0.887), Sensitivity (0.887), AP (0.834), F1 score (0.880), and AUC (0.945) of the random forest model were higher than other models.

Fig. 2

(A) ROC curve of the six ML models for predicting medical disputes. (B) Precision recall curve of the six ML models for predicting medical disputes

Table 3 AUC values and PSI for each model test set

Table 4 The results of each evaluation parameter for each model test set

Fig. 3

Calibration curves of six ML models for predicting medical disputes in the test set

Calibration curves for six machine learning model test sets showed random forest to be the closest to perfectly accurate curve with the highest goodness of fit (R² = 0.546). It was followed by bagging classifier, AdaBoost classifier, decision tree classifier, and Gaussian NB. The calibration curve for logistic regression was poor(R² = 0.022)(Fig. 3).

Analysis of clinical decision curve revealed that in the whole threshold probability range, patients benefited the most from random forest model, followed by bagging classifier model, decision tree classifier, and logistics regression. When the threshold probability of the AdaBoost classifier model was below 0.43, the curve coincided with the treat-all curve. Thus, patients benefit for the threshold probability of ˃0.43. The Gaussian NB model only benefits patients for the threshold probability of ˃0.35. The above results are detailed in Fig. 4.

Fig. 4

Decision curve analysis of the six ML models for predicting medical disputes in the test set

The inpatient department contributed the most to the dispute. In particular, Orthopedics (76.644%), Critical care medicine (73.529%), Gynecology (70.707%), Pediatrics (67.010%), Obstetrics (64.912%), General surgery (55.525%), and Cardiology (51.538%). The aforementioned departments were followed by the Hospitalization expenses, hospitalization days, discharge type, doctor title, patient age, patient occupation, payment method, and blood transfusion volume, hospitalization times, patent gender. In other words, the higher the hospitalization expenses and the longer the hospitalization days, the more probable would be the disputes. Disputes are more likely to occur after the death of older patients. The farmers and patients with employee medical insurance are prone to disputes. More disputes were associated with higher volume of blood transfusion. The importance ranking of variable features is shown in Fig. 5.

Fig. 5

Importance distribution of the random forest model

Discussion

Influencing factors in medical disputes and their contribution to disputes

In this study, departments contributed the most to the occurrence of disputes. Departments were divided into 33 sub-departments based on disease. According to analysis results, significant differences existed in the occurrence of disputes among departments. Among them, Orthopedics, Intensive Care Medicine, Rehabilitation, Gynecology, Pediatrics, Obstetrics, General Surgery, Respiratory, and Cardiology had a high incidence of disputes. This is in agreement with the findings of Pan Rong et al. [21]. Orthopedics is the department with the highest incidence of disputes. However, this study was mainly focused on inpatient departments, and medical technology departments were not considered. Rong et al. found high incidences of disputes in medical technology departments, such as radiology and ultrasound.

In addition, the more the hospitalization days and the higher the treatment cost, the more probable the disputes. The contribution of hospitalization days and treatment cost to the occurrence of disputes is second only to the department. According to some scholars, some patients interpret medical behavior as “commodity trading” and believe that the higher the cost, the better the therapeutic effect; otherwise, they cannot accept reality [22].

We also analyzed the discharge methods and found a statistical difference between following two groups: the most likely to have disputes after the death of patients and the least likely to have disputes in the way of transferring medical advice to the community. Suhua et al. believed that discharge methods could reflect medical quality and safety, and different discharge methods had different effects on the occurrence of disputes [23], which was consistent with the conclusion of this study. In addition, the contribution of discharge methods follows that of hospitalization costs.

We found the incidence of disputes among junior professional doctors to be higher than that among senior doctors. Among all investigated medical disputes, 36% were caused by deficient clinical practices among doctors. This is often because young doctors lack standardized professional training and are largely deficient in clinical practice, which diminishes service quality [24]. The contribution to the occurrence of disputes was second only to the number of days in the hospital.

Among all complaints, a higher number of female patients was involved in disputes than that of male patients, which iwas consistent with the results of other studies [25]. Gender contributed the least to the occurrence of disputes.

The occupation and payment methods of patients must also be considered by the hospitals. The farmers and non-workers were found to involve more in disputes, and the payment method was medical insurance of the residents and patients with full self-payment. The analysis indicates that people without jobs or medical insurance and with relatively low education are under greater economic pressure after illness. They believe the medical expenses to be the same as daily consumption and equate a good treatment with treatment cost. They hold hospital responsible in case of complications if any and thus, believe that they must receive economic compensation [26]. In this study, the contributions of occupation and payment method to disputes were higher than that of the gender of the patient.

In addition, the higher the amount of blood transfused, the more probable the disputes. According to Zhao et al. [27], patients placed significant expectations on blood transfusion, ignored its complexity, and wrongly believed that blood transfusion could address many problems. Doctors examined patients rapidly before transfusion, and the management of the blood transfusion department was not standardized. In this study, the contribution of blood transfusion volume to the occurrence of disputes was higher than that of patient gender and lower than that of occupation.

Numerous studies have shown that marriage, nationality, and quarter have no significant effect on the occurrence of medical disputes [28]. Our study findings were in good agreement with these previous findings.

Random forest model has a good performance to predict medical disputes

In this study, among the six ML models used, the random forest model had the best prediction efficiency for medical disputes, and the area under the ROC curve reached 0.945. The accuracy and other indicators of the model were higher than those of other models. The traditional logistic regression model had the worst performance (AUC = 0.826); however, the model was more stable. In the study of cardiovascular disease prediction model based on random forest in eastern China, Li et al. found that the random forest was superior to other methods with an AUC of 0.787. This study also found that the AUC of random forest is 0.945, which is better than other models [29].

Application of medical dispute prediction model in clinical practice

Although both ROC and DCA are key curves for model evaluation and can be used to evaluate the pros and cons of different diagnostic methods or models, essential differences are present. DCA is a simple method to evaluate clinical predictive models, diagnostic tests, and molecular markers. While traditional diagnostic test indicators such as sensitivity, specificity, and area under the ROC curve only measure the diagnostic accuracy of the predictive model and fail to consider the clinical utility of a specific model, the advantage of DCA is that it integrates the preferences of patients or decision makers into the analysis [30]. This idea meets the practical needs of clinical decision-making and is increasingly widely used in clinical analysis. Because The ROC curve only focuses on the accuracy of the model and does not focus on the relationship between the benefits and risks caused by the prediction information, the model with the maximum AUC is not necessarily optimal. The ROC and DCA curves together determine the efficiency of the model. In our study, among the six ML models used, the ROC and DCA indicated that the random forest model had the strongest classification ability and benefited the patient and doctor the most in clinical practice. Therefore, the number of disputes was significantly reduced after the intervention for all predicted positive patients, and the benefit to true positive patients was greater than the loss to false positive patients. The intervention plan primarily changes the treatment plan, strengthens the nursing level, and helps doctors focus on the patients, which makes the patients have a better medical experience and treatment effect. Simultaneously, for medical institutions, in addition to all the predicted positive patients requiring a significant amount of human resources for intervention, the reduction in disputes significantly reduces the costs of medical compensation and dispute handling and the mental pressure caused by disputes on medical staff. Thus, the medical institutions obviously are benefited.

Research limitations and prospects

The main limitation of this study was the lack of external validation. ML can be considered an internal validation to some extent because it consists of multiple data-oriented analyses by randomly splitting the data repeatedly. External validation and optimization of the current model must be performed in future studies. Second, the object of study was a sampled from only one city in China. We hope more cities to be included in the future for multi-center studies, introduce more advanced technologies in AI, and realize more functions in product development in the future.

Conclusion

The influencing factors of medical disputes and their contribution to disputes were in descending order of inpatient department, hospitalization expenses, discharge type, hospitalization days, title of the doctor, patient age, occupation, payment methods, blood transfusion volume, hospitalization times, and gender of the patient. The random forest model was found more effective than the other five models in the prediction of disputes, and the patient and the doctor benefit the most in clinical application.