To see the other types of publications on this topic, follow the link: Post-hoc interpretability.
Journal articles on the topic 'Post-hoc interpretability'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Post-hoc interpretability.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Feng, Jiangfan, Yukun Liang, and Lin Li. "Anomaly Detection in Videos Using Two-Stream Autoencoder with Post Hoc Interpretability." Computational Intelligence and Neuroscience 2021 (July 26, 2021): 1–15. http://dx.doi.org/10.1155/2021/7367870.
Full textAbstract:
The growing interest in deep learning approaches to video surveillance raises concerns about the accuracy and efficiency of neural networks. However, fast and reliable detection of abnormal events is still a challenging work. Here, we introduce a two-stream approach that offers an autoencoder-based structure for fast and efficient detection to facilitate anomaly detection from surveillance video without labeled abnormal events. Furthermore, we present post hoc interpretability of feature map visualization to show the process of feature learning, revealing uncertain and ambiguous decision boundaries in the video sequence. Experimental results on Avenue, UCSD Ped2, and Subway datasets show that our method can detect abnormal events well and explain the internal logic of the model at the object level.
2
Sinhamahapatra, Poulami, Suprosanna Shit, Anjany Sekuboyina, et al. "Enhancing Interpretability of Vertebrae Fracture Grading using Human-interpretable Prototypes." Machine Learning for Biomedical Imaging 2, July 2024 (2024): 977–1002. http://dx.doi.org/10.59275/j.melba.2024-258b.
Full textAbstract:
Vertebral fracture grading classifies the severity of vertebral fractures, which is a challenging task in medical imaging and has recently attracted Deep Learning (DL) models. Only a few works attempted to make such models human-interpretable despite the need for transparency and trustworthiness in critical use cases like DL-assisted medical diagnosis. Moreover, such models either rely on post-hoc methods or additional annotations. In this work, we propose a novel interpretable-by-design method, ProtoVerse, to find relevant sub-parts of vertebral fractures (prototypes) that reliably explain the model’s decision in a human-understandable way. Specifically, we introduce a novel diversity-promoting loss to mitigate prototype repetitions in small datasets with intricate semantics. We have experimented with the VerSe’19 dataset and outperformed the existing prototype-based method. Further, our model provides superior interpretability against the post-hoc method. Importantly, expert radiologists validated the visual interpretability of our results, showing clinical applicability.
3
Sarma Borah, Proyash Paban, Devraj Kashyap, Ruhini Aktar Laskar, and Ankur Jyoti Sarmah. "A Comprehensive Study on Explainable AI Using YOLO and Post Hoc Method on Medical Diagnosis." Journal of Physics: Conference Series 2919, no. 1 (2024): 012045. https://doi.org/10.1088/1742-6596/2919/1/012045.
Full textAbstract:
Abstract Medical imaging plays a pivotal role in disease detection and intervention. The black-box nature of deep learning models, such as YOLOv8, creates challenges in interpreting their decisions. This paper presents a toolset to enhance interpretability in AI based diagnostics by integrating Explainable AI (XAI) techniques with YOLOv8. This paper explores implementation of post hoc methods, including Grad-CAM and Eigen CAM, to assist end users in understanding the decision making of the model. This comprehensive evaluation utilises CT-Datasets, demonstrating the efficacy of YOLOv8 for object detection in different medical fields. This paper compares the interpretability offered by different post hoc methods, shedding light on abnormalities detected by the model. Moreover, this paper introduces a user-friendly interface for end users, incorporating the generated heat maps for intuitive understanding using different CAM algorithms. These findings underscore the importance of XAI in medical image analysis and offer a practical framework for improving interpretability in X-ray diagnostics. The comparison of the different CAM methods can offer a choice for end users to determine the best fit for deployable tools. This work contributes to bridging the gap between sophisticated deep learning models and actionable insights for professionals. Access at https://spritan.github.io/YOLOv8_Explainer/
4
Zhang, Zaixi, Qi Liu, Hao Wang, Chengqiang Lu, and Cheekong Lee. "ProtGNN: Towards Self-Explaining Graph Neural Networks." Proceedings of the AAAI Conference on Artificial Intelligence 36, no. 8 (2022): 9127–35. http://dx.doi.org/10.1609/aaai.v36i8.20898.
Full textAbstract:
Despite the recent progress in Graph Neural Networks (GNNs), it remains challenging to explain the predictions made by GNNs. Existing explanation methods mainly focus on post-hoc explanations where another explanatory model is employed to provide explanations for a trained GNN. The fact that post-hoc methods fail to reveal the original reasoning process of GNNs raises the need of building GNNs with built-in interpretability. In this work, we propose Prototype Graph Neural Network (ProtGNN), which combines prototype learning with GNNs and provides a new perspective on the explanations of GNNs. In ProtGNN, the explanations are naturally derived from the case-based reasoning process and are actually used during classification. The prediction of ProtGNN is obtained by comparing the inputs to a few learned prototypes in the latent space. Furthermore, for better interpretability and higher efficiency, a novel conditional subgraph sampling module is incorporated to indicate which part of the input graph is most similar to each prototype in ProtGNN+. Finally, we evaluate our method on a wide range of datasets and perform concrete case studies. Extensive results show that ProtGNN and ProtGNN+ can provide inherent interpretability while achieving accuracy on par with the non-interpretable counterparts.
5
Alfano, Gianvincenzo, Sergio Greco, Domenico Mandaglio, Francesco Parisi, Reza Shahbazian, and Irina Trubitsyna. "Even-if Explanations: Formal Foundations, Priorities and Complexity." Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 15 (2025): 15347–55. https://doi.org/10.1609/aaai.v39i15.33684.
Full textAbstract:
Explainable AI has received significant attention in recent years. Machine learning models often operate as black boxes, lacking explainability and transparency while supporting decision-making processes. Local post-hoc explainability queries attempt to answer why individual inputs are classified in a certain way by a given model. While there has been important work on counterfactual explanations, less attention has been devoted to semifactual ones. In this paper, we focus on local post-hoc explainability queries within the semifactual `even-if' thinking and their computational complexity among different classes of models, and show that both linear and tree-based models are strictly more interpretable than neural networks. After this, we introduce a preference-based framework enabling users to personalize explanations based on their preferences, both in the case of semifactuals and counterfactuals, enhancing interpretability and user-centricity. Finally, we explore the complexity of several interpretability problems in the proposed preference-based framework and provide algorithms for polynomial cases.
6
Xu, Qian, Wenzhao Xie, Bolin Liao, et al. "Interpretability of Clinical Decision Support Systems Based on Artificial Intelligence from Technological and Medical Perspective: A Systematic Review." Journal of Healthcare Engineering 2023 (February 3, 2023): 1–13. http://dx.doi.org/10.1155/2023/9919269.
Full textAbstract:
Background. Artificial intelligence (AI) has developed rapidly, and its application extends to clinical decision support system (CDSS) for improving healthcare quality. However, the interpretability of AI-driven CDSS poses significant challenges to widespread application. Objective. This study is a review of the knowledge-based and data-based CDSS literature regarding interpretability in health care. It highlights the relevance of interpretability for CDSS and the area for improvement from technological and medical perspectives. Methods. A systematic search was conducted on the interpretability-related literature published from 2011 to 2020 and indexed in the five databases: Web of Science, PubMed, ScienceDirect, Cochrane, and Scopus. Journal articles that focus on the interpretability of CDSS were included for analysis. Experienced researchers also participated in manually reviewing the selected articles for inclusion/exclusion and categorization. Results. Based on the inclusion and exclusion criteria, 20 articles from 16 journals were finally selected for this review. Interpretability, which means a transparent structure of the model, a clear relationship between input and output, and explainability of artificial intelligence algorithms, is essential for CDSS application in the healthcare setting. Methods for improving the interpretability of CDSS include ante-hoc methods such as fuzzy logic, decision rules, logistic regression, decision trees for knowledge-based AI, and white box models, post hoc methods such as feature importance, sensitivity analysis, visualization, and activation maximization for black box models. A number of factors, such as data type, biomarkers, human-AI interaction, needs of clinicians, and patients, can affect the interpretability of CDSS. Conclusions. The review explores the meaning of the interpretability of CDSS and summarizes the current methods for improving interpretability from technological and medical perspectives. The results contribute to the understanding of the interpretability of CDSS based on AI in health care. Future studies should focus on establishing formalism for defining interpretability, identifying the properties of interpretability, and developing an appropriate and objective metric for interpretability; in addition, the user's demand for interpretability and how to express and provide explanations are also the directions for future research.
7
Gill, Navdeep, Patrick Hall, Kim Montgomery, and Nicholas Schmidt. "A Responsible Machine Learning Workflow with Focus on Interpretable Models, Post-hoc Explanation, and Discrimination Testing." Information 11, no. 3 (2020): 137. http://dx.doi.org/10.3390/info11030137.
Full textAbstract:
This manuscript outlines a viable approach for training and evaluating machine learning systems for high-stakes, human-centered, or regulated applications using common Python programming tools. The accuracy and intrinsic interpretability of two types of constrained models, monotonic gradient boosting machines and explainable neural networks, a deep learning architecture well-suited for structured data, are assessed on simulated data and publicly available mortgage data. For maximum transparency and the potential generation of personalized adverse action notices, the constrained models are analyzed using post-hoc explanation techniques including plots of partial dependence and individual conditional expectation and with global and local Shapley feature importance. The constrained model predictions are also tested for disparate impact and other types of discrimination using measures with long-standing legal precedents, adverse impact ratio, marginal effect, and standardized mean difference, along with straightforward group fairness measures. By combining interpretable models, post-hoc explanations, and discrimination testing with accessible software tools, this text aims to provide a template workflow for machine learning applications that require high accuracy and interpretability and that mitigate risks of discrimination.
8
Kulaklıoğlu, Duru. "Explainable AI: Enhancing Interpretability of Machine Learning Models." Human Computer Interaction 8, no. 1 (2024): 91. https://doi.org/10.62802/z3pde490.
Full textAbstract:
Explainable Artificial Intelligence (XAI) is emerging as a critical field to address the “black box” nature of many machine learning (ML) models. While these models achieve high predictive accuracy, their opacity undermines trust, adoption, and ethical compliance in critical domains such as healthcare, finance, and autonomous systems. This research explores methodologies and frameworks to enhance the interpretability of ML models, focusing on techniques like feature attribution, surrogate models, and counterfactual explanations. By balancing model complexity and transparency, this study highlights strategies to bridge the gap between performance and explainability. The integration of XAI into ML workflows not only fosters trust but also aligns with regulatory requirements, enabling actionable insights for stakeholders. The findings reveal a roadmap to design inherently interpretable models and tools for post-hoc analysis, offering a sustainable approach to democratize AI.
9
Acun, Cagla, and Olfa Nasraoui. "Pre Hoc and Co Hoc Explainability: Frameworks for Integrating Interpretability into Machine Learning Training for Enhanced Transparency and Performance." Applied Sciences 15, no. 13 (2025): 7544. https://doi.org/10.3390/app15137544.
Full textAbstract:
Post hoc explanations for black-box machine learning models have been criticized for potentially inaccurate surrogate models and computational burden at prediction time. We propose pre hoc and co hoc explainability frameworks that integrate interpretability directly into the training process through an inherently interpretable white-box model. Pre hoc uses the white-box model to regularize the black-box model, while co hoc jointly optimizes both models with a shared loss function. We extend these frameworks to generate instance-specific explanations using Jensen–Shannon divergence as a regularization term. Our two-phase approach first trains models for fidelity, then generates local explanations through neighborhood-based fine-tuning. Experiments on credit risk scoring and movie recommendation datasets demonstrate superior global and local fidelity compared to LIME, without compromising accuracy. The co hoc framework additionally enhances white-box model accuracy by up to 3%, making it valuable for regulated domains requiring interpretable models. Our approaches provide more faithful and consistent explanations at a lower computational cost than existing methods, offering a promising direction for making machine learning models more transparent and trustworthy while maintaining high prediction accuracy.
10
Yousufi Aqmal, Shahid, and Fermle Erdely S. "Enhancing Nonparametric Tests: Insights for Computational Intelligence and Data Mining." Researcher Academy Innovation Data Analysis 1, no. 3 (2024): 214–26. https://doi.org/10.69725/raida.v1i3.168.
Full textAbstract:
Objective: With the aim of improving monitoring reliability and interpretability of CI and DM experimental statistical tests, we evaluate the performance of cutting-edge nonparametric tests and post hoc procedures. Methods: A Friedman Aligned Ranks test, Quade test, and multiple post hoc corrections Bonferroni-Dunn and Holm were used to comparative analyze data. These approaches were employed to algorithm performance metrics with varied datasets to evaluate their capability to detect meaningful differences and control Type I errors.Results: Advanced nonparametric methods consistently outperformed traditional parametric tests, offering robust results in heterogeneous datasets. The Quade test was the most powerful and stable, and the post hoc procedures greatly increased the power of the pairwise comparisons.Novelty: We evaluate advanced nonparametric methods in CI and DM experiments: the Friedman Aligned Ranks test, the Quade test, and post hoc procedures (Bonferroni-Dunn and Holm). These methods represent a departure from traditional parametric tests that depend on assumptions of normality and homogeneity of variance, allowing for more flexible and robust approaches to analyses of complex, heterogeneous datasets. By comparing the strength and efficacy of these methods, the research also delivers common guidelines for their use; as well as demonstrating their utility in realistic situations characterized by non-standard and dispersed data.Implications for Research: The findings have far-reaching theoretical and pragmatic implications for scholars in CI and DM. On a theoretical level, this work undermines the common bias towards parametric techniques, providing an increasingly robust framework for comparative analysis in experimental research. This work improves understanding of the adaptation of statistical tests to fit the complexities of real-world data by highlighting the advantages of advanced nonparametric methods, specifically the Quade test and post hoc corrections. Practical implications The results give owners of data summaries actionable recommendations, which will assist researchers in the selection of statistical methods that are tuned to the nature of their datasets, resulting in improved reliability and interpretability of future evaluations of algorithms. Thus, this endeavor will promote more powerful and statistically appropriate methods in CI and DM studies, leading to more confident and valid claims surrounding algorithmic performance.
11
Arjunan, Gopalakrishnan. "Implementing Explainable AI in Healthcare: Techniques for Interpretable Machine Learning Models in Clinical Decision-Making." International Journal of Scientific Research and Management (IJSRM) 9, no. 05 (2021): 597–603. http://dx.doi.org/10.18535/ijsrm/v9i05.ec03.
Full textAbstract:
The integration of explainable artificial intelligence (XAI) in healthcare is revolutionizing clinical decision-making by providing clarity around complex machine learning (ML) models. As AI becomes increasingly critical in medical fields—ranging from diagnostics to treatment personalization—the interpretability of these models is crucial for fostering trust, transparency, and accountability among healthcare providers and patients. Traditional "black-box" models, such as deep neural networks, often achieve high accuracy but lack transparency, creating challenges in highly regulated, high-stakes settings like healthcare. Explainable AI addresses this issue by employing methods that make model decisions understandable and justifiable, ensuring that clinicians can interpret, trust, and apply AI recommendations safely and effectively. This paper presents a comprehensive analysis of explainable AI techniques specifically tailored for healthcare applications, focusing on two primary approaches: intrinsic interpretability and post-hoc interpretability. Intrinsic techniques, which design models to be naturally interpretable (e.g., decision trees, logistic regression), enable clinicians to directly trace and understand the rationale behind model predictions. Post-hoc techniques, on the other hand, provide interpretability for complex models after they have been trained. Examples include SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and saliency maps in medical imaging, each of which provides insights into how and why specific predictions are made. This study also examines the unique challenges of implementing explainable AI in healthcare, such as balancing accuracy with interpretability, addressing the diversity of stakeholder needs, and ensuring data privacy. Through real-world case studies—such as early sepsis detection in intensive care units and the use of saliency maps in radiology—the paper demonstrates how explainable AI improves clinical workflows, enhances patient outcomes, and fosters regulatory compliance by enabling transparency in automated decision-making. Ultimately, this work underscores the transformative potential of explainable AI to make machine learning models not only powerful but also trustworthy, actionable, and ethical in the context of healthcare.
12
Khiem, Phan Xuan, Zurida B. Batchaeva, and Liana K. Katchieva. "HYBRIDIZATION OF MACHINE LEARNING AND STATISTICS METHODS TO IMPROVE MODEL INTERPRETABILITY." EKONOMIKA I UPRAVLENIE: PROBLEMY, RESHENIYA 12/15, no. 153 (2024): 214–20. https://doi.org/10.36871/ek.up.p.r.2024.12.15.025.
Full textAbstract:
The article discusses the hybridization of machine learning and statistics methods to improve the interpretability of models. Interpretability is a key factor for decision making in areas such as medicine, finance, and social sciences, where algorithm transparency is critical. The proposed approach combines the accuracy and flexibility of machine learning with the analytical capabilities of statistical methods. Integration methods are discussed, including the use of confidence intervals, Bayesian methods, principal component analysis, and post-hoc interpretation approaches such as SHAP. The results demonstrate that hybrid methods allow justifying the significance of predictors, taking into account the uncertainty of predictions, and adapting the level of explanations to the needs of different users. The advantages, limitations, and prospects for further development of hybrid approaches are discussed. The work opens up new opportunities for creating transparent and reliable models applicable to critical problems.
13
Marconato, Emanuele, Andrea Passerini, and Stefano Teso. "Interpretability Is in the Mind of the Beholder: A Causal Framework for Human-Interpretable Representation Learning." Entropy 25, no. 12 (2023): 1574. http://dx.doi.org/10.3390/e25121574.
Full textAbstract:
Research on Explainable Artificial Intelligence has recently started exploring the idea of producing explanations that, rather than being expressed in terms of low-level features, are encoded in terms of interpretable concepts learned from data. How to reliably acquire such concepts is, however, still fundamentally unclear. An agreed-upon notion of concept interpretability is missing, with the result that concepts used by both post hoc explainers and concept-based neural networks are acquired through a variety of mutually incompatible strategies. Critically, most of these neglect the human side of the problem: a representation is understandable only insofar as it can be understood by the human at the receiving end. The key challenge in human-interpretable representation learning (hrl) is how to model and operationalize this human element. In this work, we propose a mathematical framework for acquiring interpretable representations suitable for both post hoc explainers and concept-based neural networks. Our formalization of hrl builds on recent advances in causal representation learning and explicitly models a human stakeholder as an external observer. This allows us derive a principled notion of alignment between the machine’s representation and the vocabulary of concepts understood by the human. In doing so, we link alignment and interpretability through a simple and intuitive name transfer game, and clarify the relationship between alignment and a well-known property of representations, namely disentanglement. We also show that alignment is linked to the issue of undesirable correlations among concepts, also known as concept leakage, and to content-style separation, all through a general information-theoretic reformulation of these properties. Our conceptualization aims to bridge the gap between the human and algorithmic sides of interpretability and establish a stepping stone for new research on human-interpretable representations.
14
Biryukov, D. N., and A. S. Dudkin. "Explainability and interpretability are important aspects in ensuring the security of decisions made by intelligent systems (review article)." Scientific and Technical Journal of Information Technologies, Mechanics and Optics 25, no. 3 (2025): 373–86. https://doi.org/10.17586/2226-1494-2025-25-3-373-386.
Full textAbstract:
The issues of trust in decisions made (formed) by intelligent systems are becoming more and more relevant. A systematic review of Explicable Artificial Intelligence (XAI) methods and tools aimed at bridging the gap between the complexity of neural networks and the need for interpretability of results for end users is presented. A theoretical analysis of the differences between explainability and interpretability in the context of artificial intelligence as well as their role in ensuring the security of decisions made by intelligent systems is carried out. It is shown that explainability implies the ability of a system to generate justifications understandable to humans, whereas interpretability focuses on the passive clarity of internal mechanisms. A classification of XAI methods is proposed based on their approach (preliminary/subsequent analysis: ante hoc/post hoc) and the scale of explanations (local/global). Popular tools, such as Local Interpretable Model Agnostic Explanations, Shapley Values, and integrated gradients, are considered, with an assessment of their strengths and limitations of applicability. Practical recommendations are given on the choice of methods for various fields and scenarios. The architecture of an intelligent system based on the V.K. Finn model and adapted to modern requirements for ensuring “transparency” of solutions, where the key components are the information environment, the problem solver and the intelligent interface, are discussed. The problem of a compromise between the accuracy of models and their explainability is considered: transparent models (“glass boxes”, for example, decision trees) are inferior in performance to deep neural networks, but provide greater certainty of decision-making. Examples of methods and software packages for explaining and interpreting machine learning data and models are provided. It is shown that the development of XAI is associated with the integration of neuro-symbolic approaches combining deep learning capabilities with logical interpretability.
15
Gaurav, Kashyap. "Explainable AI (XAI): Methods and Techniques to Make Deep Learning Models More Interpretable and Their Real-World Implications." International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences 11, no. 4 (2023): 1–7. https://doi.org/10.5281/zenodo.14382747.
Full textAbstract:
The goal of the developing field of explainable artificial intelligence (XAI) is to make complex AI models, especially deep learning (DL) models, which are frequently criticized for being "black boxes" more interpretable. Understanding how deep learning models make decisions is becoming crucial for accountability, fairness, and trust as deep learning is used more and more in various industries. This paper offers a thorough analysis of the strategies and tactics used to improve the interpretability of deep learning models, including hybrid approaches, post-hoc explanations, and model-specific strategies. We examine the trade-offs between interpretability, accuracy, and computational complexity and draw attention to the difficulties in applying XAI in high-stakes domains like autonomous systems, healthcare, and finance. The study concludes by outlining the practical applications of XAI, such as how it affects ethical AI implementation, regulatory compliance, and decision-making.
16
Larriva-Novo, Xavier, Luis Pérez Miguel, Victor A. Villagra, Manuel Álvarez-Campana, Carmen Sanchez-Zas, and Óscar Jover. "Post-Hoc Categorization Based on Explainable AI and Reinforcement Learning for Improved Intrusion Detection." Applied Sciences 14, no. 24 (2024): 11511. https://doi.org/10.3390/app142411511.
Full textAbstract:
The massive usage of Internet services nowadays has led to a drastic increase in cyberattacks, including sophisticated techniques, so that Intrusion Detection Systems (IDSs) need to use AP technologies to enhance their effectiveness. However, this has resulted in a lack of interpretability and explainability from different applications that use AI predictions, making it hard to understand by cybersecurity operators why decisions were made. To address this, the concept of Explainable AI (XAI) has been introduced to make the AI’s decisions more understandable at both global and local levels. This not only boosts confidence in the AI but also aids in identifying different attributes commonly used in cyberattacks for the exploitation of flaws or vulnerabilities. This study proposes two developments: first, the creation and evaluation of machine learning models for an IDS with the objective to use Reinforcement Learning (RL) to classify malicious network traffic, and second, the development of a methodology to extract multi-level explanations from the RL model to identify, detect, and understand how different attributes affect uncertain types of attack categories.
17
Degtiarova, Ganna, Fran Mikulicic, Jan Vontobel, et al. "Post-hoc motion correction for coronary computed tomography angiography without additional radiation dose - Improved image quality and interpretability for “free”." Imaging 14, no. 2 (2022): 82–88. http://dx.doi.org/10.1556/1647.2022.00060.
Full textAbstract:
AbstractObjectiveTo evaluate the impact of a motion-correction (MC) algorithm, applicable post-hoc and not dependent on extended padding, on the image quality and interpretability of coronary computed tomography angiography (CCTA).MethodsNinety consecutive patients undergoing CCTA on a latest-generation 256-slice CT device were prospectively included. CCTA was performed with prospective electrocardiogram-triggering and the shortest possible acquisition window (without padding) at 75% of the R-R-interval. All datasets were reconstructed without and with MC of the coronaries. The latter exploits the minimal padding inherent in cardiac CT scans with this device due to data acquisition also during the short time interval needed for the tube to reach target currents and voltage (“free” multiphase). Two blinded readers independently assessed image quality on a 4-point Likert scale for all segments.ResultsA total of 1,030 coronary segments were evaluated. Application of MC both with automatic and manual coronary centerline tracking resulted in a significant improvement in image quality as compared to the standard reconstruction without MC (mean Likert score 3.67 [3.50;3.81] vs 3.58 [3.40;3.73], P = 0.005, and 3.7 [3.55;3.82] vs 3.58 [3.40;3.73], P < 0.001, respectively). Furthermore, MC significantly reduced the proportion of non-evaluable segments and patients with at least one non-evaluable coronary segment from 2% to as low as 0.3%, and from 14% to as low as 3%. Reduction of motion artifacts was predominantly observed in the right coronary artery.ConclusionsA post-hoc device-specific MC algorithm improves image quality and interpretability of prospectively electrocardiogram-triggered CCTA and reduces the proportion of non-evaluable scans without any additional radiation dose exposure.
18
Domen, Mohorčič, and Ocepek David. "[Re] Hierarchical Shrinkage: Improving the Accuracy and Interpretability of Tree-Based Methods." ReScience C 9, no. 2 (2023): #19. https://doi.org/10.5281/zenodo.8173696.
Full text
19
Jishnu, Setia. "Explainable AI: Methods and Applications." Explainable AI: Methods and Applications 8, no. 10 (2023): 5. https://doi.org/10.5281/zenodo.10021461.
Full textAbstract:
Explainable Artificial Intelligence (XAI) has emerged as a critical area of research, ensuring that AI systems are transparent, interpretable, and accountable. This paper provides a comprehensive overview of various methods and applications of Explainable AI. We delve into the importance of interpretability in AI models, explore different techniques for making complex AI models understandable, and discuss real-world applications where explainability is crucial. Through this paper, I aim to shed light on the advancements in the field of XAI and its potentialto bridge the gap between AI's predictions and human understanding.Keywords:- Explainable AI (XAI), Interpretable Machine Learning, Transparent AI, AI Transparency, Interpretability in AI, Ethical AI, Explainable Machine Learning Models, Model Transparency, AI Accountability, Trustworthy AI, AI Ethics, XAI Techniques, LIME (Local Interpretable Model- agnostic Explanations), SHAP (SHapley Additive exPlanations), Rule-based Explanation, Post-hoc Explanation, AI and Society, Human-AI Collaboration, AI Regulation, Trust in Artificial Intelligence.
20
Lao, Danning, Qi Liu, Jiazi Bu, Junchi Yan, and Wei Shen. "ViTree: Single-Path Neural Tree for Step-Wise Interpretable Fine-Grained Visual Categorization." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 3 (2024): 2866–73. http://dx.doi.org/10.1609/aaai.v38i3.28067.
Full textAbstract:
As computer vision continues to advance and finds widespread applications across various domains, the need for interpretability in deep learning models becomes paramount. Existing methods often resort to post-hoc techniques or prototypes to explain the decision-making process, which can be indirect and lack intrinsic illustration. In this research, we introduce ViTree, a novel approach for fine-grained visual categorization that combines the popular vision transformer as a feature extraction backbone with neural decision trees. By traversing the tree paths, ViTree effectively selects patches from transformer-processed features to highlight informative local regions, thereby refining representations in a step-wise manner. Unlike previous tree-based models that rely on soft distributions or ensembles of paths, ViTree selects a single tree path, offering a clearer and simpler decision-making process. This patch and path selectivity enhances model interpretability of ViTree, enabling better insights into the model's inner workings. Remarkably, extensive experimentation validates that this streamlined approach surpasses various strong competitors and achieves state-of-the-art performance while maintaining exceptional interpretability which is proved by multi-perspective methods. Code can be found at https://github.com/SJTU-DeepVisionLab/ViTree.
21
Zhou, Pei-Yuan, Amane Takeuchi, Fernando Martinez-Lopez, Malikeh Ehghaghi, Andrew K. C. Wong, and En-Shiun Annie Lee. "Benchmarking Interpretability in Healthcare Using Pattern Discovery and Disentanglement." Bioengineering 12, no. 3 (2025): 308. https://doi.org/10.3390/bioengineering12030308.
Full textAbstract:
The healthcare industry seeks to integrate AI into clinical applications, yet understanding AI decision making remains a challenge for healthcare practitioners as these systems often function as black boxes. Our work benchmarks the Pattern Discovery and Disentanglement (PDD) system’s unsupervised learning algorithm, which provides interpretable outputs and clustering results from clinical notes to aid decision making. Using the MIMIC-IV dataset, we process free-text clinical notes and ICD-9 codes with Term Frequency-Inverse Document Frequency and Topic Modeling. The PDD algorithm discretizes numerical features into event-based features, discovers association patterns from a disentangled statistical feature value association space, and clusters clinical records. The output is an interpretable knowledge base linking knowledge, patterns, and data to support decision making. Despite being unsupervised, PDD demonstrated performance comparable to supervised deep learning models, validating its clustering ability and knowledge representation. We benchmark interpretability techniques—Feature Permutation, Gradient SHAP, and Integrated Gradients—on the best-performing models (in terms of F1, ROC AUC, balanced accuracy, etc.), evaluating these based on sufficiency, comprehensiveness, and sensitivity metrics. Our findings highlight the limitations of feature importance ranking and post hoc analysis for clinical diagnosis. Meanwhile, PDD’s global interpretability effectively compensates for these issues, helping healthcare practitioners understand the decision-making process and providing suggestive clusters of diseases to assist their diagnosis.
22
Ganguly, Rita, Dharmpal Singh, and Rajesh Bose. "The next frontier of explainable artificial intelligence (XAI) in healthcare services: A study on PIMA diabetes dataset." Scientific Temper 16, no. 05 (2025): 4165–70. https://doi.org/10.58414/scientifictemper.2025.16.5.01.
Full textAbstract:
The integration of Artificial Intelligence (AI) in healthcare has revolutionized disease diagnosis and risk prediction. However, the "black-box" nature of AI models raises concerns about trust, interpretability, and regulatory compliance. Explainable AI (XAI) addresses these issues by enhancing transparency in AI-driven decisions. This study explores the role of XAI in diabetes prediction using the PIMA Diabetes Dataset, evaluating machine learning models—logistic regression, decision trees, random forests, and deep learning—alongside SHAP and LIME explainability techniques. Data pre-processing includes handling missing values, feature scaling, and selection. Model performance is assessed through accuracy, AUC-ROC, precision-recall, F1-score, and computational efficiency. Findings reveal that the Random Forest model achieved the highest accuracy (93%) but required post-hoc explainability. Logistic Regression provided inherent interpretability but with lower accuracy (81%). SHAP identified glucose, BMI, and age as key diabetes predictors, offering robust global explanations at a higher computational cost. LIME, with lower computational overhead, provided localized insights but lacked comprehensive interpretability. SHAP’s exponential complexity limits real-time deployment, while LIME’s linear complexity makes it more practical for clinical decision support.These insights underscore the importance of XAI in enhancing transparency and trust in AI-driven healthcare. Integrating explainability techniques can improve clinical decision-making and regulatory compliance. Future research should focus on hybrid XAI models that optimize accuracy, interpretability, and computational efficiency for real-time deployment in healthcare settings.
23
Naresh Vurukonda. "A Novel Framework for Inherently Interpretable Deep Neural Networks Using Attention-Based Feature Attribution in High-Dimensional Tabular Data." Journal of Information Systems Engineering and Management 10, no. 50s (2025): 599–604. https://doi.org/10.52783/jisem.v10i50s.10290.
Full textAbstract:
Deep learning models for tabular data often lack interpretability, posing challenges in domains like healthcare and finance where trust is critical. We propose an attention-augmented neural network architecture that inherently highlights the most informative features, thus providing intrinsic explanations for its predictions. Drawing inspiration from TabNet and Transformer-based models, our model applies multi-head feature-wise attention to automatically weight each feature’s contribution. We incorporate an attention-weight regularization scheme (e.g. sparsemax) to encourage focused attributions. For further interpretability, we compare these learned attention weights with SHAP (Shapley Additive Explanations) post-hoc values. We evaluate our approach on a high-dimensional healthcare dataset (e.g. clinical outcome prediction) and synthetic benchmarks. Experimental results show our model achieves competitive accuracy (Table 1) while providing clear feature importance insights. Feature attribution charts (Fig. 1) demonstrate that the attention mechanism successfully identifies key predictors, aligning well with SHAP analysis. This work bridges performance and explainability by design, enabling reliable deployment of deep models on complex tabular data.
24
Naresh Vurukonda. "A Novel Framework for Inherently Interpretable Deep Neural Networks Using Attention-Based Feature Attribution in High-Dimensional Tabular Data." Journal of Information Systems Engineering and Management 10, no. 51s (2025): 1076–81. https://doi.org/10.52783/jisem.v10i51s.10626.
Full textAbstract:
Deep learning models for tabular data often lack interpretability, posing challenges in domains like healthcare and finance where trust is critical. We propose an attention-augmented neural network architecture that inherently highlights the most informative features, thus providing intrinsic explanations for its predictions. Drawing inspiration from TabNet and Transformer-based models, our model applies multi-head feature-wise attention to automatically weight each feature’s contribution. We incorporate an attention-weight regularization scheme (e.g. sparsemax) to encourage focused attributions. For further interpretability, we compare these learned attention weights with SHAP (Shapley Additive Explanations) post-hoc values. We evaluate our approach on a high-dimensional healthcare dataset (e.g. clinical outcome prediction) and synthetic benchmarks. Experimental results show our model achieves competitive accuracy (Table 1) while providing clear feature importance insights. Feature attribution charts (Fig. 1) demonstrate that the attention mechanism successfully identifies key predictors, aligning well with SHAP analysis. This work bridges performance and explainability by design, enabling reliable deployment of deep models on complex tabular data.
25
Ozdemir, Olcar. "Explainable AI (XAI) in Healthcare: Bridging the Gap between Accuracy and Interpretability." Journal of Science, Technology and Engineering Research 1, no. 1 (2024): 32–44. https://doi.org/10.64206/0z78ev10.
Full textAbstract:
Artificial Intelligence (AI) has demonstrated significant potential in revolutionizing healthcare by enhancing diagnostic accuracy, predicting patient outcomes, and optimizing treatment plans. However, the increasing reliance on complex, black-box models has raised critical concerns around transparency, trust, and accountability—particularly in high-stakes medical settings where interpretability is vital for clinical decision-making. This paper explores Explainable AI (XAI) as a solution to bridge the gap between model performance and human interpretability. We review current XAI techniques, including post-hoc methods like SHAP and LIME, and intrinsically interpretable models, assessing their applicability and limitations within healthcare contexts. Through selected case studies in radiology, oncology, and clinical decision support systems, we examine how XAI can improve clinician trust and facilitate informed decision-making without compromising predictive accuracy. Our analysis highlights persistent challenges such as balancing explanation fidelity with usability, addressing data biases, and aligning explanations with clinical reasoning. We propose a multidisciplinary framework that integrates technical, ethical, and user-centered principles to support the development of trustworthy XAI systems. Future research directions include the standardization of interpretability metrics, the co-design of models with clinicians, and regulatory considerations for deploying XAI in clinical practice. By aligning technological advances with human-centered design, XAI has the potential to transform AI into a reliable partner in healthcare delivery.
26
García-Vicente, Clara, David Chushig-Muzo, Inmaculada Mora-Jiménez, et al. "Evaluation of Synthetic Categorical Data Generation Techniques for Predicting Cardiovascular Diseases and Post-Hoc Interpretability of the Risk Factors." Applied Sciences 13, no. 7 (2023): 4119. http://dx.doi.org/10.3390/app13074119.
Full textAbstract:
Machine Learning (ML) methods have become important for enhancing the performance of decision-support predictive models. However, class imbalance is one of the main challenges for developing ML models, because it may bias the learning process and the model generalization ability. In this paper, we consider oversampling methods for generating synthetic categorical clinical data aiming to improve the predictive performance in ML models, and the identification of risk factors for cardiovascular diseases (CVDs). We performed a comparative study of several categorical synthetic data generation methods, including Synthetic Minority Oversampling Technique Nominal (SMOTEN), Tabular Variational Autoencoder (TVAE) and Conditional Tabular Generative Adversarial Networks (CTGANs). Then, we assessed the impact of combining oversampling strategies and linear and nonlinear supervised ML methods. Lastly, we conducted a post-hoc model interpretability based on the importance of the risk factors. Experimental results show the potential of GAN-based models for generating high-quality categorical synthetic data, yielding probability mass functions that are very close to those provided by real data, maintaining relevant insights, and contributing to increasing the predictive performance. The GAN-based model and a linear classifier outperform other oversampling techniques, improving the area under the curve by 2%. These results demonstrate the capability of synthetic data to help with both determining risk factors and building models for CVD prediction.
27
Jalali, Anahid, Alexander Schindler, Bernhard Haslhofer, and Andreas Rauber. "Machine Learning Interpretability Techniques for Outage Prediction: A Comparative Study." PHM Society European Conference 5, no. 1 (2020): 10. http://dx.doi.org/10.36001/phme.2020.v5i1.1244.
Full textAbstract:
Interpretable machine learning has recently attracted a lot of interest in the community. Currently, it mainly focuses on models trained on non-time series data. LIME and SHAP are well-known examples and provide visual-explanations of feature contributions to model decisions on an instance basis. Other post-hoc approaches, such as attribute-wise interpretations, also focus on tabular data only. Little research has been done so far on the interpretability of predictive models trained on time series data. Therefore, this work focuses on explaining decisions made by black-box models such as Deep Neural Networks trained on sensor data. In this paper, we present the results of a qualitative study, in which we systematically compare the types of explanations and the properties (e.g., method, computational complexity) of existing interpretability approaches for models trained on the PHM08-CMAPSS dataset. We compare shallow models such as regression trees (with limited depth) and black-box models such as Long-Short Term Memories (LSTMs) and Support Vector Regression (SVR). We train models on processed sensor data and explain their output using LIME, SHAP, and attribute-wise methods. Throughout our experiments, we point out the advantages and disadvantages of using these approaches for interpreting models trained on time series data. Our investigation results can serve as a guideline for selecting a suitable explainability method for black-box predictive models trained on time-series data.
28
LE, Khanh Giang. "IMPROVING ROAD SAFETY: SUPERVISED MACHINE LEARNING ANALYSIS OF FACTORS INFLUENCING CRASH SEVERITY." Scientific Journal of Silesian University of Technology. Series Transport 127 (June 1, 2025): 129–53. https://doi.org/10.20858/sjsutst.2025.127.8.
Full textAbstract:
Road traffic crash severity is shaped by a complex interplay of human, vehicular, environmental, and infrastructural factors. While machine learning (ML) has shown promise in analyzing crash data, gaps remain in model interpretability and region-specific insights, particularly for the UK context. This study addresses these gaps by evaluating supervised ML models – Decision Tree, Support Vector Machine (SVM), and LightGBM – to predict crash severity using 2022 UK accident data. The research emphasizes interpretability through SHapley Additive exPlanations (SHAP) to identify critical factors influencing severity outcomes. Results demonstrate that LightGBM outperforms other models in predictive performance, with police officer attendance at the scene, speed limits, and the number of vehicles involved emerging as pivotal determinants of severity. The analysis reveals that higher speed limits and single-vehicle collisions correlate with severe outcomes, while police presence may mitigate accident severity. However, the study acknowledges limitations, including dataset constraints. By integrating ML with post-hoc interpretability techniques, this work advances actionable insights for policymakers to prioritize road safety interventions, such as optimizing enforcement strategies and revising speed regulations. The findings underscore the potential of interpretable ML frameworks to enhance understanding of crash dynamics and inform targeted safety measures, contributing to global efforts to reduce traffic-related fatalities and injuries.
29
Zdravkovic, Milan. "On the global feature importance for interpretable and trustworthy heat demand forecasting." Thermal Science, no. 00 (2025): 48. https://doi.org/10.2298/tsci241223048z.
Full textAbstract:
The paper introduces the Explainable AI methodology to assess the global feature importance of the Machine Learning models used for heat demand forecasting in intelligent control of District Heating Systems (DHS), with motivation to facilitate their interpretability and trustworthiness, hence addressin g the challenges related to adherence to communal standards, customer satisfaction and liability risks. Methodology involves generation of global feature importance insights by using four different approaches, namely intrinsic (ante-hoc) interpretability of Gradient Boosting method and selected post-hoc methods, namely Partial Dependence, Accumulated Local Effects (ALE) and SHAP and qualitative analysis of those insights in context of expected behavior of DHS and comparative analysis. None of the selected methods assume feature permutation or perturbations which can introduce bias due to introduction of random unrealistic values of data instances. ALE and SHAP have been found as most reliable methods for determining the feature importance, taking into account feature interactions and nonlinearities. ALE plots with transmitted energy across the range of ambient temperatures closely resemble the shape of the control curve, which is the evidence of accurate model, as well as suitability of explanation method. By providing the insights which align with the domain expertise, the discussion confirms the value of using Explainable AI stack as mandatory layer in assessing the performance of ML models, especially in high-risk AI systems, such as those whose use is anticipated in the DHS.
30
Gunasekara, Sachini, and Mirka Saarela. "Explainable AI in Education: Techniques and Qualitative Assessment." Applied Sciences 15, no. 3 (2025): 1239. https://doi.org/10.3390/app15031239.
Full textAbstract:
Many of the articles on AI in education compare the performance and fairness of different models, but few specifically focus on quantitatively analyzing their explainability. To bridge this gap, we analyzed key evaluation metrics for two machine learning models—ANN and DT—with a focus on their performance and explainability in predicting student outcomes using the OULAD. The methodology involved evaluating the DT, an intrinsically explainable model, against the more complex ANN, which requires post hoc explainability techniques. The results show that, although the feature-based and structured decision-making process of the DT facilitates natural interpretability, it struggles to model complex data relationships, often leading to misclassification. In contrast, the ANN demonstrated higher accuracy and stability but lacked transparency. Crucially, the ANN showed great fidelity in result predictions when it used the LIME and SHAP methods. The results of the experiments verify that the ANN consistently outperformed the DT in prediction accuracy and stability, especially with the LIME method. However, improving the interpretability of ANN models remains a challenge for future research.
31
Maddala, Suresh Kumar. "Understanding Explainability in Enterprise AI Models." International Journal of Management Technology 12, no. 1 (2025): 58–68. https://doi.org/10.37745/ijmt.2013/vol12n25868.
Full textAbstract:
This article examines the critical role of explainability in enterprise AI deployments, where algorithmic transparency has emerged as both a regulatory necessity and a business imperative. As organizations increasingly rely on sophisticated machine learning models for consequential decisions, the "black box" problem threatens stakeholder trust, regulatory compliance, and effective model governance. We explore the multifaceted business case for explainable AI across regulated industries, analyze the spectrum of interpretability techniques—from inherently transparent models to post-hoc explanation methods for complex neural networks—and investigate industry-specific applications in finance, healthcare, fraud detection, and human resources. The article addresses practical implementation challenges, including the accuracy-interpretability tradeoff, computational constraints, and ethical considerations around data bias. Looking forward, the article examines emerging developments in regulatory frameworks, hybrid model architectures, causal inference approaches, and integrated explanation interfaces. Throughout the analysis, the article demonstrates that explainability is not merely a technical consideration but a foundational element of responsible AI deployment that allows organizations to balance innovation with accountability in an increasingly algorithm-driven business landscape.
32
Ali, Ali Mohammed Omar. "Explainability in AI: Interpretable Models for Data Science." International Journal for Research in Applied Science and Engineering Technology 13, no. 2 (2025): 766–71. https://doi.org/10.22214/ijraset.2025.66968.
Full textAbstract:
As artificial intelligence (AI) continues to drive advancements across various domains, the need for explainability in AI models has become increasingly critical. Many state-of-the-art machine learning models, particularly deep learning architectures, operate as "black boxes," making their decision-making processes difficult to interpret. Explainable AI (XAI) aims to enhance model transparency, ensuring that AI-driven decisions are understandable, trustworthy, and aligned with ethical and regulatory standards. This paper explores different approaches to AI interpretability, including intrinsically interpretable models such as decision trees and logistic regression, as well as post-hoc methods like SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-Agnostic Explanations). Additionally, we discuss the challenges of explainability, including the trade-off between accuracy and interpretability, scalability issues, and domain-specific requirements. The paper also highlights real-world applications of XAI in healthcare, finance, and autonomous systems. Finally, we examine future research directions, emphasizing hybrid models, causal explainability, and human-AI collaboration. By fostering more interpretable AI systems, we can enhance trust, fairness, and accountability in data science applications.
33
Vinayak Pillai. "Enhancing the transparency of data and ml models using explainable AI (XAI)." World Journal of Advanced Engineering Technology and Sciences 13, no. 1 (2024): 397–406. http://dx.doi.org/10.30574/wjaets.2024.13.1.0428.
Full textAbstract:
To this end, this paper focuses on the increasing demand for the explainability of Machine Learning (ML) models especially in environments where these models are employed to make critical decisions such as in healthcare, finance, and law. Although the typical ML models are considered opaque, XAI provides a set of ways and means to propose making these models more transparent and, thus, easier to explain. This paper describes and analyzes the model-agnostic approach, method of intrinsic explanation, post-hoc explanation, and visualization instruments and demonstrates the use of XAI in various fields. The paper also speaks about the requirement of capturing the accuracy and interpretability for creating responsible and ethical AI.
34
Wang, Zhengguang. "Validation, Robustness, and Accuracy of Perturbation-Based Sensitivity Analysis Methods for Time-Series Deep Learning Models." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 21 (2024): 23768–70. http://dx.doi.org/10.1609/aaai.v38i21.30559.
Full textAbstract:
This work undertakes studies to evaluate Interpretability Methods for Time Series Deep Learning. Sensitivity analysis assesses how input changes affect the output, constituting a key component of interpretation. Among the post-hoc interpretation methods such as back-propagation, perturbation, and approximation, my work will investigate perturbation-based sensitivity Analysis methods on modern Transformer models to benchmark their performances. Specifically, my work intends to answer three research questions: 1) Do different sensitivity analysis methods yield comparable outputs and attribute importance rankings? 2) Using the same sensitivity analysis method, do different Deep Learning models impact the output of the sensitivity analysis? 3) How well do the results from sensitivity analysis methods align with the ground truth?
35
Chatterjee, Soumick, Arnab Das, Chirag Mandal, et al. "TorchEsegeta: Framework for Interpretability and Explainability of Image-Based Deep Learning Models." Applied Sciences 12, no. 4 (2022): 1834. http://dx.doi.org/10.3390/app12041834.
Full textAbstract:
Clinicians are often very sceptical about applying automatic image processing approaches, especially deep learning-based methods, in practice. One main reason for this is the black-box nature of these approaches and the inherent problem of missing insights of the automatically derived decisions. In order to increase trust in these methods, this paper presents approaches that help to interpret and explain the results of deep learning algorithms by depicting the anatomical areas that influence the decision of the algorithm most. Moreover, this research presents a unified framework, TorchEsegeta, for applying various interpretability and explainability techniques for deep learning models and generates visual interpretations and explanations for clinicians to corroborate their clinical findings. In addition, this will aid in gaining confidence in such methods. The framework builds on existing interpretability and explainability techniques that are currently focusing on classification models, extending them to segmentation tasks. In addition, these methods have been adapted to 3D models for volumetric analysis. The proposed framework provides methods to quantitatively compare visual explanations using infidelity and sensitivity metrics. This framework can be used by data scientists to perform post hoc interpretations and explanations of their models, develop more explainable tools, and present the findings to clinicians to increase their faith in such models. The proposed framework was evaluated based on a use case scenario of vessel segmentation models trained on Time-of-Flight (TOF) Magnetic Resonance Angiogram (MRA) images of the human brain. Quantitative and qualitative results of a comparative study of different models and interpretability methods are presented. Furthermore, this paper provides an extensive overview of several existing interpretability and explainability methods.
36
Sai Teja Boppiniti. "A SURVEY ON EXPLAINABLE AI: TECHNIQUES AND CHALLENGES." International Journal of Innovations in Engineering Research and Technology 7, no. 3 (2020): 57–66. http://dx.doi.org/10.26662/ijiert.v7i3.pp57-66.
Full textAbstract:
Explainable Artificial Intelligence (XAI) is a rapidly evolving field aimed at making AI systems more interpretable and transparent to human users. As AI technologies become increasingly integrated into critical sectors such as healthcare, finance, and autonomous systems, the need for explanations behind AI decisions has grown significantly. This survey provides a comprehensive review of XAI techniques, categorizing them into post-hoc and intrinsic methods, and examines their application in various domains. Additionally, the paper explores the major challenges in achieving explainability, including balancing accuracy with interpretability, scalability, and the trade-off between transparency and complexity. The survey concludes with a discussion on the future directions of XAI, emphasizing the importance of interdisciplinary approaches to developing robust and interpretable AI systems.
37
Murdoch, W. James, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. "Definitions, methods, and applications in interpretable machine learning." Proceedings of the National Academy of Sciences 116, no. 44 (2019): 22071–80. http://dx.doi.org/10.1073/pnas.1900654116.
Full textAbstract:
Machine-learning models have demonstrated great success in learning complex patterns that enable them to make predictions about unobserved data. In addition to using models for prediction, the ability to interpret what a model has learned is receiving an increasing amount of attention. However, this increased focus has led to considerable confusion about the notion of interpretability. In particular, it is unclear how the wide array of proposed interpretation methods are related and what common concepts can be used to evaluate them. We aim to address these concerns by defining interpretability in the context of machine learning and introducing the predictive, descriptive, relevant (PDR) framework for discussing interpretations. The PDR framework provides 3 overarching desiderata for evaluation: predictive accuracy, descriptive accuracy, and relevancy, with relevancy judged relative to a human audience. Moreover, to help manage the deluge of interpretation methods, we introduce a categorization of existing techniques into model-based and post hoc categories, with subgroups including sparsity, modularity, and simulatability. To demonstrate how practitioners can use the PDR framework to evaluate and understand interpretations, we provide numerous real-world examples. These examples highlight the often underappreciated role played by human audiences in discussions of interpretability. Finally, based on our framework, we discuss limitations of existing methods and directions for future work. We hope that this work will provide a common vocabulary that will make it easier for both practitioners and researchers to discuss and choose from the full range of interpretation methods.
38
Aslam, Nida, Irfan Ullah Khan, Samiha Mirza, et al. "Interpretable Machine Learning Models for Malicious Domains Detection Using Explainable Artificial Intelligence (XAI)." Sustainability 14, no. 12 (2022): 7375. http://dx.doi.org/10.3390/su14127375.
Full textAbstract:
With the expansion of the internet, a major threat has emerged involving the spread of malicious domains intended by attackers to perform illegal activities aiming to target governments, violating privacy of organizations, and even manipulating everyday users. Therefore, detecting these harmful domains is necessary to combat the growing network attacks. Machine Learning (ML) models have shown significant outcomes towards the detection of malicious domains. However, the “black box” nature of the complex ML models obstructs their wide-ranging acceptance in some of the fields. The emergence of Explainable Artificial Intelligence (XAI) has successfully incorporated the interpretability and explicability in the complex models. Furthermore, the post hoc XAI model has enabled the interpretability without affecting the performance of the models. This study aimed to propose an Explainable Artificial Intelligence (XAI) model to detect malicious domains on a recent dataset containing 45,000 samples of malicious and non-malicious domains. In the current study, initially several interpretable ML models, such as Decision Tree (DT) and Naïve Bayes (NB), and black box ensemble models, such as Random Forest (RF), Extreme Gradient Boosting (XGB), AdaBoost (AB), and Cat Boost (CB) algorithms, were implemented and found that XGB outperformed the other classifiers. Furthermore, the post hoc XAI global surrogate model (Shapley additive explanations) and local surrogate LIME were used to generate the explanation of the XGB prediction. Two sets of experiments were performed; initially the model was executed using a preprocessed dataset and later with selected features using the Sequential Forward Feature selection algorithm. The results demonstrate that ML algorithms were able to distinguish benign and malicious domains with overall accuracy ranging from 0.8479 to 0.9856. The ensemble classifier XGB achieved the highest result, with an AUC and accuracy of 0.9991 and 0.9856, respectively, before the feature selection algorithm, while there was an AUC of 0.999 and accuracy of 0.9818 after the feature selection algorithm. The proposed model outperformed the benchmark study.
39
Roscher, R., B. Bohn, M. F. Duarte, and J. Garcke. "EXPLAIN IT TO ME – FACING REMOTE SENSING CHALLENGES IN THE BIO- AND GEOSCIENCES WITH EXPLAINABLE MACHINE LEARNING." ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences V-3-2020 (August 3, 2020): 817–24. http://dx.doi.org/10.5194/isprs-annals-v-3-2020-817-2020.
Full textAbstract:
Abstract. For some time now, machine learning methods have been indispensable in many application areas. Especially with the recent development of efficient neural networks, these methods are increasingly used in the sciences to obtain scientific outcomes from observational or simulated data. Besides a high accuracy, a desired goal is to learn explainable models. In order to reach this goal and obtain explanation, knowledge from the respective domain is necessary, which can be integrated into the model or applied post-hoc. We discuss explainable machine learning approaches which are used to tackle common challenges in the bio- and geosciences, such as limited amount of labeled data or the provision of reliable and scientific consistent results. We show that recent advances in machine learning to enhance transparency, interpretability, and explainability are helpful in overcoming these challenges.
40
Damilare Tiamiyu, Seun Oluwaremilekun Aremu, Igba Emmanuel, Chidimma Judith Ihejirika, Michael Babatunde Adewoye, and Adeshina Akin Ajayi. "Interpretable Data Analytics in Blockchain Networks Using Variational Autoencoders and Model-Agnostic Explanation Techniques for Enhanced Anomaly Detection." International Journal of Scientific Research in Science and Technology 11, no. 6 (2024): 152–83. http://dx.doi.org/10.32628/ijsrst24116170.
Full textAbstract:
The rapid growth of blockchain technology has brought about increased transaction volumes and complexity, leading to challenges in detecting fraudulent activities and understanding data patterns. Traditional data analytics approaches often fall short in providing both accurate anomaly detection and interpretability, especially in decentralized environments. This paper explores the integration of Variational Autoencoders (VAEs), a deep learning-based anomaly detection technique, with model-agnostic explanation methods such as SHAP (SHapley Additive Explanations) and LIME (Local Interpretable Model-agnostic Explanations) to enhance the interpretability of blockchain data analytics. Variational Autoencoders are leveraged to capture the underlying distribution of blockchain transactions, identifying anomalies by modeling deviations from learned patterns. To address the often-opaque nature of deep learning models, SHAP and LIME are employed to provide post-hoc explanations, offering insights into the key factors influencing the model’s predictions. This hybrid approach aims to not only detect irregularities in blockchain networks effectively but also to make the decision-making process transparent and understandable for stakeholders. By combining advanced anomaly detection with interpretable machine learning, this study presents a robust framework for improving the security and reliability of blockchain-based systems, providing a valuable tool for both developers and analysts in mitigating risks and enhancing trust in decentralized applications.
41
Guo, Jiaxing, Zhiyi Tang, Changxing Zhang, Wei Xu, and Yonghong Wu. "An Interpretable Deep Learning Method for Identifying Extreme Events under Faulty Data Interference." Applied Sciences 13, no. 9 (2023): 5659. http://dx.doi.org/10.3390/app13095659.
Full textAbstract:
Structural health monitoring systems continuously monitor the operational state of structures, generating a large amount of monitoring data during the process. The structural responses of extreme events, such as earthquakes, ship collisions, or typhoons, could be captured and further analyzed. However, it is challenging to identify these extreme events due to the interference of faulty data. Real-world monitoring systems suffer from frequent misidentification and false alarms. Unfortunately, it is difficult to improve the system’s built-in algorithms, especially the deep neural networks, partly because the current neural networks only output results and do not provide an interpretable decision-making basis. In this study, a deep learning-based method with visual interpretability is proposed to identify seismic data under sensor faults interference. The transfer learning technique is employed to learn the features of seismic data and faulty data with efficiency. A post hoc interpretation algorithm, termed Gradient-weighted Class Activation Mapping (Grad-CAM), is embedded into the neural networks to uncover the interest regions that support the output decision. The in situ seismic responses of a cable-stayed long-span bridge are used for method verification. The results show that the proposed method can effectively identify seismic data mixed with various types of faulty data while providing good interpretability.
42
Methuku, Vijayalaxmi, Sharath Chandra Kondaparthy, and Direesh Reddy Aunugu. "Explainability and Transparency in Artificial Intelligence: Ethical Imperatives and Practical Challenges." International Journal of Electrical, Electronics and Computers 8, no. 3 (2023): 7–12. https://doi.org/10.22161/eec.84.2.
Full textAbstract:
Artificial Intelligence (AI) is increasingly embedded in high-stakes domains such as healthcare, finance, and law enforcement, where opaque decision-making raises significant ethical concerns. Among the core challenges in AI ethics are explainability and transparency—key to fostering trust, accountability, and fairness in algorithmic systems. This review explores the ethical foundations of explainable AI (XAI), surveys leading technical approaches such as model-agnostic interpretability techniques and post-hoc explanation methods and examines their inherent limitations and trade-offs. A real-world case study from the healthcare sector highlights the critical consequences of deploying non-transparent AI models in clinical decision-making. The article also discusses emerging regulatory frameworks and underscores the need for interdisciplinary collaboration to address the evolving ethical landscape. The review concludes with recommendations for aligning technical innovation with ethical imperatives through responsible design and governance.
43
Qian, Wei, Chenxu Zhao, Yangyi Li, Fenglong Ma, Chao Zhang, and Mengdi Huai. "Towards Modeling Uncertainties of Self-Explaining Neural Networks via Conformal Prediction." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 13 (2024): 14651–59. http://dx.doi.org/10.1609/aaai.v38i13.29382.
Full textAbstract:
Despite the recent progress in deep neural networks (DNNs), it remains challenging to explain the predictions made by DNNs. Existing explanation methods for DNNs mainly focus on post-hoc explanations where another explanatory model is employed to provide explanations. The fact that post-hoc methods can fail to reveal the actual original reasoning process of DNNs raises the need to build DNNs with built-in interpretability. Motivated by this, many self-explaining neural networks have been proposed to generate not only accurate predictions but also clear and intuitive insights into why a particular decision was made. However, existing self-explaining networks are limited in providing distribution-free uncertainty quantification for the two simultaneously generated prediction outcomes (i.e., a sample's final prediction and its corresponding explanations for interpreting that prediction). Importantly, they also fail to establish a connection between the confidence values assigned to the generated explanations in the interpretation layer and those allocated to the final predictions in the ultimate prediction layer. To tackle the aforementioned challenges, in this paper, we design a novel uncertainty modeling framework for self-explaining networks, which not only demonstrates strong distribution-free uncertainty modeling performance for the generated explanations in the interpretation layer but also excels in producing efficient and effective prediction sets for the final predictions based on the informative high-level basis explanations. We perform the theoretical analysis for the proposed framework. Extensive experimental evaluation demonstrates the effectiveness of the proposed uncertainty framework.
44
Okajima, Yuzuru, and Kunihiko Sadamasa. "Deep Neural Networks Constrained by Decision Rules." Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 17, 2019): 2496–505. http://dx.doi.org/10.1609/aaai.v33i01.33012496.
Full textAbstract:
Deep neural networks achieve high predictive accuracy by learning latent representations of complex data. However, the reasoning behind their decisions is difficult for humans to understand. On the other hand, rule-based approaches are able to justify the decisions by showing the decision rules leading to them, but they have relatively low accuracy. To improve the interpretability of neural networks, several techniques provide post-hoc explanations of decisions made by neural networks, but they cannot guarantee that the decisions are always explained in a simple form like decision rules because their explanations are generated after the decisions are made by neural networks.In this paper, to balance the accuracy of neural networks and the interpretability of decision rules, we propose a hybrid technique called rule-constrained networks, namely, neural networks that make decisions by selecting decision rules from a given ruleset. Because the networks are forced to make decisions based on decision rules, it is guaranteed that every decision is supported by a decision rule. Furthermore, we propose a technique to jointly optimize the neural network and the ruleset from which the network select rules. The log likelihood of correct classifications is maximized under a model with hyper parameters about the ruleset size and the prior probabilities of rules being selected. This feature makes it possible to limit the ruleset size or prioritize human-made rules over automatically acquired rules for promoting the interpretability of the output. Experiments on datasets of time-series and sentiment classification showed rule-constrained networks achieved accuracy as high as that achieved by original neural networks and significantly higher than that achieved by existing rule-based models, while presenting decision rules supporting the decisions.
45
Sateesh Kumar Rongali. "Enhancing machine learning models: addressing challenges and future directions." World Journal of Advanced Research and Reviews 25, no. 1 (2025): 1749–53. https://doi.org/10.30574/wjarr.2025.25.1.0190.
Full textAbstract:
Machine learning is considered as a core of modern artificial intelligence with progressive advancements throughout a spectrum including but not limited to healthcare and finance, natural language processing and self-driving cars. However, several problems remain to affect the efficiency, equal opportunities of users, and adaptability of ML models for an even faster-growing era. The limitations include shortage of high quality and access to training data, model complexity that can lead to overfitting, built in bias of the algorithm, interpretability and finally, the computational density needed for such big data models. These problems posed challenges to translate the knowledge derived from the ML systems into real-world use as well as hindering generalization of ML systems, particularly the medical and legal fields that have requirements of fairness and interpretability. These are the basic issues this journal addresses and provides possible ways of enhancing the performance of the ML models. To mitigate the problem of data deficiency, we present various techniques including data augmentation and transfer learning. To mitigate this issue, we present regularization strategies and methods of model validation. Several prevention methods are also mentioned including biasing of Algorithm and models using adversarial biasing, and Fairness-aware learning methods. Furthermore, we explore the increasing relevance of post-hoc model interpretability such as the SHAP and LIME methods which explain model’s outputs in a more detailed manner. The objective of the present Journal is to support further development of more stable, efficient and fair Machine Learning systems. In this paper, recent developments and long-term solutions are discussed to prepare the way for better and more responsible use of AI in the future.
46
Huai, Mengdi, Jinduo Liu, Chenglin Miao, Liuyi Yao, and Aidong Zhang. "Towards Automating Model Explanations with Certified Robustness Guarantees." Proceedings of the AAAI Conference on Artificial Intelligence 36, no. 6 (2022): 6935–43. http://dx.doi.org/10.1609/aaai.v36i6.20651.
Full textAbstract:
Providing model explanations has gained significant popularity recently. In contrast with the traditional feature-level model explanations, concept-based explanations can provide explanations in the form of high-level human concepts. However, existing concept-based explanation methods implicitly follow a two-step procedure that involves human intervention. Specifically, they first need the human to be involved to define (or extract) the high-level concepts, and then manually compute the importance scores of these identified concepts in a post-hoc way. This laborious process requires significant human effort and resource expenditure due to manual work, which hinders their large-scale deployability. In practice, it is challenging to automatically generate the concept-based explanations without human intervention due to the subjectivity of defining the units of concept-based interpretability. In addition, due to its data-driven nature, the interpretability itself is also potentially susceptible to malicious manipulations. Hence, our goal in this paper is to free human from this tedious process, while ensuring that the generated explanations are provably robust to adversarial perturbations. We propose a novel concept-based interpretation method, which can not only automatically provide the prototype-based concept explanations but also provide certified robustness guarantees for the generated prototype-based explanations. We also conduct extensive experiments on real-world datasets to verify the desirable properties of the proposed method.
47
Xue, Mufan, Xinyu Wu, Jinlong Li, Xuesong Li, and Guoyuan Yang. "A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 6 (2024): 6413–21. http://dx.doi.org/10.1609/aaai.v38i6.28461.
Full textAbstract:
Recently, convolutional neural networks (CNNs) have become the best quantitative encoding models for capturing neural activity and hierarchical structure in the ventral visual pathway. However, the weak interpretability of these black-box models hinders their ability to reveal visual representational encoding mechanisms. Here, we propose a convolutional neural network interpretable framework (CNN-IF) aimed at providing a transparent interpretable encoding model for the ventral visual pathway. First, we adapt the feature-weighted receptive field framework to train two high-performing ventral visual pathway encoding models using large-scale functional Magnetic Resonance Imaging (fMRI) in both goal-driven and data-driven approaches. We find that network layer-wise predictions align with the functional hierarchy of the ventral visual pathway. Then, we correspond feature units to voxel units in the brain and successfully quantify the alignment between voxel responses and visual concepts. Finally, we conduct Network Dissection along the ventral visual pathway including the fusiform face area (FFA), and discover variations related to the visual concept of `person'. Our results demonstrate the CNN-IF provides a new perspective for understanding encoding mechanisms in the human ventral visual pathway, and the combination of ante-hoc interpretable structure and post-hoc interpretable approaches can achieve fine-grained voxel-wise correspondence between model and brain. The source code is available at: https://github.com/BIT-YangLab/CNN-IF.
48
Kumar, Akshi, Shubham Dikshit, and Victor Hugo C. Albuquerque. "Explainable Artificial Intelligence for Sarcasm Detection in Dialogues." Wireless Communications and Mobile Computing 2021 (July 2, 2021): 1–13. http://dx.doi.org/10.1155/2021/2939334.
Full textAbstract:
Sarcasm detection in dialogues has been gaining popularity among natural language processing (NLP) researchers with the increased use of conversational threads on social media. Capturing the knowledge of the domain of discourse, context propagation during the course of dialogue, and situational context and tone of the speaker are some important features to train the machine learning models for detecting sarcasm in real time. As situational comedies vibrantly represent human mannerism and behaviour in everyday real-life situations, this research demonstrates the use of an ensemble supervised learning algorithm to detect sarcasm in the benchmark dialogue dataset, MUStARD. The punch-line utterance and its associated context are taken as features to train the eXtreme Gradient Boosting (XGBoost) method. The primary goal is to predict sarcasm in each utterance of the speaker using the chronological nature of a scene. Further, it is vital to prevent model bias and help decision makers understand how to use the models in the right way. Therefore, as a twin goal of this research, we make the learning model used for conversational sarcasm detection interpretable. This is done using two post hoc interpretability approaches, Local Interpretable Model-agnostic Explanations (LIME) and Shapley Additive exPlanations (SHAP), to generate explanations for the output of a trained classifier. The classification results clearly depict the importance of capturing the intersentence context to detect sarcasm in conversational threads. The interpretability methods show the words (features) that influence the decision of the model the most and help the user understand how the model is making the decision for detecting sarcasm in dialogues.
49
Fan, Yongxian, Meng Liu, and Guicong Sun. "An interpretable machine learning framework for diagnosis and prognosis of COVID-19." PLOS ONE 18, no. 9 (2023): e0291961. http://dx.doi.org/10.1371/journal.pone.0291961.
Full textAbstract:
Coronaviruses have affected the lives of people around the world. Increasingly, studies have indicated that the virus is mutating and becoming more contagious. Hence, the pressing priority is to swiftly and accurately predict patient outcomes. In addition, physicians and patients increasingly need interpretability when building machine models in healthcare. We propose an interpretable machine framework(KISM) that can diagnose and prognose patients based on blood test datasets. First, we use k-nearest neighbors, isolated forests, and SMOTE to pre-process the original blood test datasets. Seven machine learning tools Support Vector Machine, Extra Tree, Random Forest, Gradient Boosting Decision Tree, eXtreme Gradient Boosting, Logistic Regression, and ensemble learning were then used to diagnose and predict COVID-19. In addition, we used SHAP and scikit-learn post-hoc interpretability to report feature importance, allowing healthcare professionals and artificial intelligence models to interact to suggest biomarkers that some doctors may have missed. The 10-fold cross-validation of two public datasets shows that the performance of KISM is better than that of the current state-of-the-art methods. In the diagnostic COVID-19 task, an AUC value of 0.9869 and an accuracy of 0.9787 were obtained, and ultimately Leukocytes, platelets, and Proteina C reativa mg/dL were found to be the most indicative biomarkers for the diagnosis of COVID-19. An AUC value of 0.9949 and an accuracy of 0.9677 were obtained in the prognostic COVID-19 task and Age, LYMPH, and WBC were found to be the most indicative biomarkers for identifying the severity of the patient.
50
Ajkuna MUJO. "Explainable AI in Credit Scoring: Improving Transparency in Loan Decisions." Journal of Information Systems Engineering and Management 10, no. 27s (2025): 506–15. https://doi.org/10.52783/jisem.v10i27s.4437.
Full textAbstract:
The increasing dependence on Artificial Intelligence (AI) in the realm of credit scoring has led to notable enhancements in loan approval processes, particularly with regard to accuracy, efficiency, and risk evaluation. Yet, due to the opacity of sophisticated AI models, there are worries regarding transparency, fairness, and adherence to regulations. Because traditional black-box models like deep learning and ensemble methods are not interpretable, financial institutions find it challenging to justify credit decisions based on them. This absence of transparency creates difficulties in complying with regulatory standards such as Basel III, the Fair Lending Act, and GDPR, while also heightening the risk of biased or unjust lending practices. This study examines the role of Explainable AI (XAI) in credit scoring to tackle these issues, concentrating on methods that improve model interpretability while maintaining predictive performance. This study puts forward a credit scoring framework driven by XAI, which combines Shapley Additive Explanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME) to enhance the transparency of AI-based loan decision-making. Machine learning models such as random forests, gradient boosting, and neural networks are evaluated for their accuracy and explainability using real-world credit risk datasets. The results demonstrate that although AI improves risk prediction, post-hoc interpretability techniques effectively identify the key factors affecting loan approvals, thereby promoting trust and adherence to regulations. This research emphasizes how XAI can reduce bias, enhance fairness, and foster transparency in credit decision-making. These developments open the door to more ethical and accountable AI-based financial systems.