The Clinical Benefit of Fused AI Models: Improving PE Mortality Prediction Over Traditional Scores

The Clinical Benefit of Fused AI Models: Improving PE Mortality Prediction Over Traditional Scores

Pulmonary Embolism (PE) remains a significant cause of cardiovascular mortality, presenting a critical diagnostic and prognostic challenge in clinical practice. Accurately assessing disease severity and predicting mortality risk for patients with acute PE is paramount for guiding timely and appropriate management strategies. While traditional risk stratification tools like the Pulmonary Embolism Severity Index (PESI) have been invaluable, their limitations in providing precise, individualized prognoses underscore the urgent need for more advanced and robust predictive models.

The advent of Artificial Intelligence (AI) and deep learning (DL) offers a transformative opportunity to overcome these hurdles. By integrating diverse data types—from complex imaging scans to comprehensive clinical features—AI-driven fusion models are emerging as powerful tools that can not only enhance diagnostic accuracy but, more importantly, revolutionize prognostic assessment. This article delves into the clinical benefits of these fused AI models, demonstrating their superior capability in predicting PE mortality compared to conventional scoring systems, and outlining their potential to usher in a new era of personalized medicine.

The Challenge of Pulmonary Embolism Prognosis

Diagnosing pulmonary embolism is often just the initial step; the subsequent critical phase involves determining disease severity and predicting a patient’s short and long-term outlook. This stratification dictates treatment intensity, ranging from outpatient management to aggressive interventions for high-risk cases. For decades, clinicians have relied on tools such as the PESI score, which factors in various clinical parameters to categorize patients into different risk groups for 30-day morbidity and mortality.

PESI is a well-validated instrument, widely adopted and referenced in numerous studies for its utility in predicting outcomes. However, its effectiveness has known boundaries. A significant limitation highlighted by research is its positive predictive value for high-risk patients, which stands at only 11%. This suggests that while PESI is good at identifying those not at high risk, it frequently overestimates risk for a substantial portion of those flagged as high-risk, leading to potential overtreatment or misallocation of resources. This inherent imprecision creates a significant gap in personalized patient care, motivating the search for more granular and accurate prognostic tools.

Unlocking Deeper Insights with Multimodal AI Fusion

Recognizing the limitations of traditional scoring systems, researchers have increasingly turned to AI to augment and refine prognostic capabilities. Initial AI efforts in PE focused heavily on detection and diagnosis, showing promising results in identifying emboli on imaging scans. However, the true frontier lies in prognostication—predicting a patient’s trajectory post-diagnosis. The power of AI truly shines when it can synthesize information from multiple, disparate sources, a concept known as multimodal fusion.

Multimodal AI models combine different types of data, such as quantitative imaging features extracted from CT Pulmonary Angiography (CTPA) scans, alongside detailed clinical characteristics like patient demographics, comorbidities, and laboratory results. This comprehensive approach allows AI to identify subtle patterns and interactions that are imperceptible to human analysis or simpler statistical models. By integrating these diverse data streams, fusion models create a holistic picture of a patient’s condition, leading to more precise and robust predictions.

The profound advantage of this approach has been articulated in recent research, underscoring its potential to transform patient stratification. A foundational understanding of this methodology and its empirical validation highlights the compelling rationale for its adoption in clinical settings. The detailed discussion from a relevant study provides critical insights:

Table of Links

• Abstract
• Introduction
• Methods
• Results
• Discussion
• Conclusions, Acknowledgments, and References

4. Discussion

4.1. Rationale for Study Approach

Prior studies have explored the potential of AI-based models in PE assessment, focusing on improving detection and diagnosis of PE.[11,25-27] Once a diagnosis of acute PE has been made, determining disease severity is important to guiding clinical management. PESI is a well-validated risk assessment tool for prediction of 30-day morbidity and mortality, commonly used in clinical practice.[5] In a meta-analysis including 71 studies and 44,298 patients, PESI and simplified PESI tools were the most highly-validated models available.[28] However, PESI’s positive predictive value for high-risk patients is only 11%.[6] Considering PESI’s limitations, we sought to develop AI-based models to build upon existing tools and improve prognostication.

While other studies have shown great potential for AI in detecting and diagnosing PE, few have shown benefits of AI for prognostication. Additionally, incorporation of multimodal data allows for more heterogeneous analysis. Somani et al. supported that use of fusion models may outperform non-fusion models in PE detection, supporting our efforts to assess efficacy of different permutations of fusion models in prognostication of PE.[25] A recent study explored the use of a multimodal model for PE risk stratification, based on prediction of thirtyday all-cause mortality.[29] A deep neural network (TabNet) was combined with a CNN, relying on a single binary label for each CTPA scan. Their fusion model achieved higher performance (AUC: 0.96) compared to clinical (0.87) and imaging (0.82) models. Our study further supports how multimodal approaches can improve healthcare decision-making and prognostication in PE patients.

4.2. Discussion of Study Findings

In this study, we showed that DL models incorporating combined imaging and clinical features can achieve high performance in predicting PE mortality, improving performance over PESI alone. The multimodal model outperformed both imaging and clinical models, indicating enhanced robustness from combining imaging and clinical data. The PESI-fused model slightly outperformed the multimodal model, indicating marginal benefit from incorporating the PESI framework. Models were also compared to RSF, with RSF outperforming the imaging model, clinical model, and PESI on the internal test set. However, RSF outperformed only the imaging model on the external test set. On both internal and external test sets, RSF was outperformed by the deep multimodal and PESI-fused models, demonstrating benefits of deep multimodal learning over a traditional survival method.

Given that PESI estimates the risk of 30-day mortality, additional survival comparison was conducted to evaluate 30-day performance. PESI demonstrated greater performance in predicting short-term PE survival compared to long-term, consistent with its clinical purpose. The clinical, multimodal, and PESI-fused models demonstrated improved performance in short-term prediction compared to long-term on the internal test set. However, they demonstrated lower performance compared to long-term on the external test set. Despite the improved performance of PESI in short-term prediction, the majority of DL models still demonstrated higher performance. On internal testing, clinical, multimodal, and PESI-fused models achieved higher c-indices than PESI. On external testing, PESI outperformed the clinical model but underperformed the multimodal and PESI-fused models. These findings indicate PESI’s performance is improved in short-term prediction. However, the deep multimodal and PESI-fused models still demonstrate improved performance, subject to model generalizability. This provides insight into how model performance may vary based on the specific outcome being assessed, as short-term mortality may be influenced less by competing risk factors.

For the clinical survival prediction model, we identified the predictive ability and importance of each feature. Age and history of cancer were found to have the greatest predictive ability. History of cancer had the greatest feature importance. This analysis may provide valuable insight into the underlying mechanisms or risk factors related to the predicted outcome. The alignment of our survival prediction model with observations in clinical practice provides further validation of model rationality.

NRI was used to analyze the contributions of different modalities to the multimodal framework, as well as the contribution of PESI to the PESI-fused model, by measuring the accuracy improvement achieved by incorporating each. The NRI values for +Clinical and +Imaging were positive, indicating improved performance from the incorporation of clinical and imaging data in the multimodal framework. Meanwhile, the values for +PESI were negative/lower, indicating less of a contribution. This suggests that the integration of imaging and clinical variables provides valuable and complementary information for survival prediction, resulting in more refined and reliable classification of individuals. Much of the information within PESI is already included in clinical variables, but conflicting characterization performance of PESI may lessen its contribution to the PESI-fused model.

Given the importance of RV dysfunction as a risk factor in PE patients, an additional factor-risk analysis was performed with the multimodal survival predictions. The multimodal survival model identified 68.8% of RV dysfunction patients as high-risk. The model also demonstrated a high correlation between high-risk identification and mortality, identifying 84.6% of mortality patients as high-risk. Through this risk stratification, the survival model was shown to be capable of predicting mortality, as well as having a relatively strong correlation with the prognostic factor of RV dysfunction. Thus, our model validates the association between RV dysfunction and death in PE patients.

4.3. Limitations

There are several limitations to this study. Like most DL-based survival analysis models, there is a concern for generalizability given that the model was trained using limited data from a single institution. To ensure external validity and generalizability of our models, we trained and validated them first on the single-institution internal dataset, then tested their accuracy on the previously-unseen multi-institution external dataset. Additionally, concatenation was used to fuse the two survival prediction branches- a more effective feature fusion mechanism between imaging and clinical data remains to be investigated. As we did not have access to data regarding patient treatment strategies within hospitals, we were not able to take clustering in treatment approaches into account. We were not able to compare the predictive value of the models between different settings (inpatient, emergency department, outpatient). Lastly, due to the CTPA requirement within our inclusion criteria, our study excludes more severe cases and perhaps the majority of PE mortality as these patients typically do not survive long enough to undergo CTPA.

Before being widely accepted, our models will likely require additional validation on larger and more diverse datasets, as well as prospective testing of the developed models. With the application of DL into medical care, appropriate and robust regulatory measures must be passed, and radiologists/clinicians will need to be trained to implement such models into their workflows.

Authors:

• (1) Zhusi Zhong, BS, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA, Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA, and School of Electronic Engineering, Xidian University, Xi’an 710071, China;
• (2) Helen Zhang, BS, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA;
• (3) Fayez H. Fayad, BA, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA;
• (4) Andrew C. Lancaster, BS, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA and Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA;
• (5) John Sollee, BS, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA;
• (6) Shreyas Kulkarni, BS, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA;
• (7) Cheng Ting Lin, MD, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA;
• (8) Jie Li, PhD, School of Electronic Engineering, Xidian University, Xi’an 710071, China;
• (9) Xinbo Gao, PhD, School of Electronic Engineering, Xidian University, Xi’an 710071, China;
• (10) Scott Collins, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA;
• (11) Colin Greineder, MD, Department of Pharmacology, Medical School, University of Michigan, Ann Arbor, MI, 48109, USA;
• (12) Sun H. Ahn, MD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA;
• (13) Harrison X. Bai, MD, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA;
• (14) Zhicheng Jiao, PhD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA;
• (15) Michael K. Atalay, MD, PhD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA.

This paper is available on arxiv under CC BY 4.0 DEED license.

This discussion highlights that deep learning models, particularly those combining imaging and clinical features, significantly outperform traditional methods like PESI in predicting PE mortality. The multimodal approach demonstrates enhanced robustness, and even a “PESI-fused” model, which integrates the PESI framework into an AI model, shows marginal benefits. Crucially, factors like age and history of cancer emerged as key predictors, aligning with known clinical observations and validating the model’s rationality. The ability of these models to accurately identify high-risk patients, especially those with RV dysfunction, underscores their clinical relevance.

Tangible Improvements: AI Outperforms Traditional Scores

The research unequivocally demonstrates that deep learning models, by harnessing combined imaging and clinical features, achieve superior performance in predicting PE mortality compared to relying on PESI alone. Specifically, a multimodal model consistently outperformed both models based solely on imaging or clinical data, underscoring the synergistic benefits of integrated information. Furthermore, a PESI-fused model, which cleverly incorporates the PESI framework into the AI architecture, showed slight additional benefits, indicating that even existing validated tools can be enhanced through deep learning integration.

Comparisons against traditional survival methods, such as Random Survival Forest (RSF), further solidified the advantage of deep multimodal learning. While RSF showed some competitive performance on internal test sets, the deep multimodal and PESI-fused models consistently outperformed it on both internal and external validation sets. This robust performance across different datasets is vital for establishing the generalizability and reliability of these advanced AI tools in varied clinical environments.

The models also offered nuanced insights into short-term versus long-term mortality prediction. Although PESI is specifically designed for 30-day mortality, the deep learning models, particularly the multimodal and PESI-fused versions, still demonstrated superior or comparable performance in short-term prediction. This suggests that AI can deliver refined prognostic accuracy even within the timeframe typically covered by traditional scores, while also offering potential for longer-term outlooks.

A crucial aspect of AI model development is understanding which features drive predictions. In the clinical survival prediction model, age and a history of cancer were identified as the most powerful predictive factors, with cancer history having the greatest overall feature importance. This alignment with established clinical understanding not only validates the AI model’s insights but also offers valuable guidance for clinicians in focusing on critical risk factors. Moreover, an analysis of Right Ventricular (RV) dysfunction, a known prognostic factor in PE, revealed that the multimodal model effectively identified a significant percentage of RV dysfunction patients as high-risk, establishing a strong correlation between its predictions and patient mortality.

Despite these significant advancements, it’s important to acknowledge study limitations. Like many deep learning models, the generalizability can be a concern, as initial training often relies on data from a single institution. Efforts to validate models on diverse, external datasets are crucial for wider acceptance. Future research will also explore more sophisticated feature fusion mechanisms and account for variables not accessible in this study, such as specific patient treatment strategies, to further refine predictive accuracy. Importantly, models currently requiring CTPA scans might exclude the most severe PE cases, as these patients might not survive long enough for imaging, indicating a need for future models to integrate pre-imaging clinical data for initial risk assessment of critically ill patients.

Integrating AI for Better Patient Outcomes

The clear superior performance of fused AI models in predicting PE mortality represents a significant leap forward in patient care. By providing more accurate and personalized risk stratification, these models empower clinicians to make more informed decisions, tailor treatment plans, and optimize resource allocation. This shift can lead to reduced mortality, improved patient outcomes, and a more efficient healthcare system.

Actionable Steps Towards AI Integration:

Real-World Example:

Consider a patient presenting to the emergency department with suspected PE. A traditional PESI score might categorize them as “intermediate risk,” leading to a standard treatment protocol. However, a fused AI model, analyzing their CTPA images, blood work, and clinical history (including subtle signs of RV strain or specific comorbidities), could identify them as “high-intermediate risk” with an elevated probability of early deterioration. This more precise AI prediction could prompt immediate escalation of care, closer monitoring, or a more aggressive initial treatment, potentially preventing an adverse event and saving their life, where traditional scores might have led to a less urgent approach.