The scope and complexity of artificial intelligence (AI) applications in auditing have grown beyond automating tasks to performing decision-making tasks. Consequently, understanding how AI-based models arrive at their decisions has become crucial, particularly for auditing tasks that demand greater accountability and that involve complex decision-making processes. In this paper, we explore the implementation of explainable AI (XAI) through a fraud detection use case and demonstrate how integrating an explainability layer using XAI can improve the interpretability of AI models, enabling stakeholders to understand the models’ decision-making process. We also present emerging AI regulations in this context.

Keywords:
artificial intelligence, transparent AI, explainable AI (XAI), fraud detection, AI regulations

The integration of artificial intelligence (AI) technologies has been a revolutionary and cautious advancement in the auditing field. According to PwC (2022), the adoption of AI has the potential to drive a 14 percent increase in global GDP by 2030, making it the largest commercial opportunity in today’s fast-changing economy. AI encompasses a wide range of techniques designed to enable computer systems to replicate human intelligence and typically operates by analyzing large amounts of labeled training data to build models that can be used to predict future conditions. Although AI offers great opportunities, organizations are becoming increasingly aware of its potential risks and challenges, ranging from lack of transparency and low interpretability to lack of reproducibility and narrow validity (Islam, Eberle, Ghafoor, and Ahmed 2021; Lipton 2018; Miller 2019). AI systems are often referred to as “black-box” systems in which inputs and outputs are known, but the inner workings of the model are not readily known to end-users. The lack of transparency and interpretability of the AI model can make it difficult to understand how it makes its predictions and the underlying decision-making process that leads to this specific prediction, which is more problematic in auditing applications where there is a need for greater transparency and accountability. Moreover, AI models can be challenging to reproduce, making it difficult for auditors to verify their results. Minor alterations in the data or algorithms can result in vastly different outcomes, undermining the reproducibility required for audit verification. Furthermore, AI models may have limited validity, implying that they may not perform well on new and unseen data. This limitation becomes even more critical when considering the auditing field, where accuracy, transparency, and reliability are crucial. To address these challenges, it is important to develop more transparent and interpretable AI models. This approach aims to align AI advancements with the core principles of auditing and strengthen auditors’ trust in the use of AI technologies. Ideally, auditors’ trust in AI will also be validated by auditors’ regulators.

Given the risks associated with AI and the need to develop accountable and trustworthy AI, the field of XAI has emerged with the aim of producing more explainable and interpretable models (Turek 2018). XAI is a subdiscipline of AI that focuses on demystifying AI’s complex algorithms to increase the interpretability, transparency, fairness, and trustworthiness of AI models while maintaining a high level of performance and accuracy. A common trade-off often exists between the performance and transparency of AI models. As the accuracy of the model improves, it tends to become less transparent (Došilović, Brčić, and Hlupić 2018). The goal of XAI is to increase the interpretability of AI models and ensure that decisions made by AI models, as well as the data driving those decisions, can be explained to stakeholders in nontechnical language (Adadi and Berrada 2018).

Zhang, Cho, and Vasarhelyi (2022) argued that it is crucial to provide XAI literacy to audit professionals owing to the existing knowledge gap. Very few studies have linked state-of-the-art XAI methods to auditing practice, and there is limited research on the application of XAI methods to AI-based audit applications. In this study, we address this knowledge gap by investigating the implementation of XAI in a fraud detection context. We demonstrate that integrating an explainability layer using XAI methods can provide valuable insights into a model’s predictions and the reasoning behind its decisions, thus contributing to the understanding of the model’s predictions. This is particularly critical for auditors who rely on the veracity and clarity of AI findings to make informed decisions.

To fully explore the potential of XAI methods, we developed an AI-based fraud detection classification model in Python that can classify observations into fraud or nonfraud cases. The model was developed using an ensemble learning1 algorithm, specifically the random forest2 method (Breiman 2001), and trained on the dataset compiled by Bao et al. (2020). The dataset compiled by Bao et al. (2020) consists of financial accounting data from the Compustat fundamental annual database for fiscal years 1991–2014, encompassing 146,045 data sample observations. The observations were labeled as fraudulent or nonfraudulent based on material accounting misstatements disclosed in the SEC’s Accounting and Auditing Enforcement Releases, as documented in the dataset compiled by Bao et al. (2020). Table 1 presents a list of 28 accounting variables (Panel A), 14 financial ratio variables (Panel B), and a snapshot of the dataset (Panel C).

TABLE 1

List of Accounting and Financial Variables Sourced from the Dataset Compiled by Bao et al. (2020) 

Panel A: 28 Accounting Variables
VariablesDescriptionSource
CUR_ASSETS Current assets, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
AP Account payable, trade Bao et al. (2020); Dechow et al. (2011)  
ASSETS Assets, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
CEQ Common/ordinary equity, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
C_ST_INVS Cash and short-term investments Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
COGS Cost of goods sold Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
CS_O/S Common shares outstanding Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
DEBT_CL Debt in current liabilities, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
LT_DEBT_I Long-term debt issuance Bao et al. (2020); Dechow et al. (2011)  
LT_DEBT Long-term debt, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
DEP_AMORT Depreciation and amortization Bao et al. (2020); Cecchini et al. (2010)  
IBEI Income before extraordinary items Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
INVT Inventories, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
INVS_ADV Investment and advances, other Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
ST_INVS Short-term investments, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
CUR_LIAB Current liabilities, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
LIAB Liabilities, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
NI Net income (loss) Bao et al. (2020); Cecchini et al. (2010)  
PPE Property, plant, and equipment, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
PS Preferred/preference stock (capital), total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
RE Retained earnings Bao et al. (2020); Cecchini et al. (2010)  
REC Receivables, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
NET_SALES Sales/turnover (net) Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
SALE_CS_PS Sale of common and preferred stock Bao et al. (2020); Dechow et al. (2011)  
TAXES_P Income taxes payable Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
TAXES Income taxes, total Bao et al. (2020); Cecchini et al. (2010)  
INT_EXP Interest and related expense, total Bao et al. (2020); Cecchini et al. (2010)  
PRICE_CLOSE Price close, annual, fiscal Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
Panel A: 28 Accounting Variables
VariablesDescriptionSource
CUR_ASSETS Current assets, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
AP Account payable, trade Bao et al. (2020); Dechow et al. (2011)  
ASSETS Assets, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
CEQ Common/ordinary equity, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
C_ST_INVS Cash and short-term investments Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
COGS Cost of goods sold Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
CS_O/S Common shares outstanding Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
DEBT_CL Debt in current liabilities, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
LT_DEBT_I Long-term debt issuance Bao et al. (2020); Dechow et al. (2011)  
LT_DEBT Long-term debt, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
DEP_AMORT Depreciation and amortization Bao et al. (2020); Cecchini et al. (2010)  
IBEI Income before extraordinary items Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
INVT Inventories, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
INVS_ADV Investment and advances, other Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
ST_INVS Short-term investments, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
CUR_LIAB Current liabilities, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
LIAB Liabilities, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
NI Net income (loss) Bao et al. (2020); Cecchini et al. (2010)  
PPE Property, plant, and equipment, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
PS Preferred/preference stock (capital), total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
RE Retained earnings Bao et al. (2020); Cecchini et al. (2010)  
REC Receivables, total Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
NET_SALES Sales/turnover (net) Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
SALE_CS_PS Sale of common and preferred stock Bao et al. (2020); Dechow et al. (2011)  
TAXES_P Income taxes payable Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
TAXES Income taxes, total Bao et al. (2020); Cecchini et al. (2010)  
INT_EXP Interest and related expense, total Bao et al. (2020); Cecchini et al. (2010)  
PRICE_CLOSE Price close, annual, fiscal Bao et al. (2020); Cecchini et al. (2010); Dechow et al. (2011)  
Panel B: 14 Financial Ratio Variables
VariablesDescriptionSource
WC_ACC WC accruals Bao et al. (2020); Dechow et al. (2011)  
RSST_ACC RSST accruals Bao et al. (2020); Dechow et al. (2011)  
CH_REC Change in receivables Bao et al. (2020); Dechow et al. (2011)  
CH_INV Change in inventory Bao et al. (2020); Dechow et al. (2011)  
% SOFT_ASSETS % soft assets Bao et al. (2020); Dechow et al. (2011)  
DEP_INDEX Depreciation index Bao et al. (2020); Beneish (1999)  
CH_CSH_SALES Change in cash sales Bao et al. (2020); Dechow et al. (2011)  
CH_CSH_MG Change in cash margin Bao et al. (2020); Dechow et al. (2011)  
CH_ROA Change in return on assets Bao et al. (2020); Dechow et al. (2011)  
CH_FCF Change in free cash flows Bao et al. (2020); Dechow et al. (2011)  
RE/ASSETS Retained earnings over total assets Bao et al. (2020); Summers and Sweeney (1998)  
EBIT/ASSETS Earnings before interest and taxes over total assets Bao et al. (2020); Summers and Sweeney (1998)  
ISSUE Actual issuance Bao et al. (2020); Dechow et al. (2011)  
B/M Book-to-market Bao et al. (2020); Dechow et al. (2011)  
Panel B: 14 Financial Ratio Variables
VariablesDescriptionSource
WC_ACC WC accruals Bao et al. (2020); Dechow et al. (2011)  
RSST_ACC RSST accruals Bao et al. (2020); Dechow et al. (2011)  
CH_REC Change in receivables Bao et al. (2020); Dechow et al. (2011)  
CH_INV Change in inventory Bao et al. (2020); Dechow et al. (2011)  
% SOFT_ASSETS % soft assets Bao et al. (2020); Dechow et al. (2011)  
DEP_INDEX Depreciation index Bao et al. (2020); Beneish (1999)  
CH_CSH_SALES Change in cash sales Bao et al. (2020); Dechow et al. (2011)  
CH_CSH_MG Change in cash margin Bao et al. (2020); Dechow et al. (2011)  
CH_ROA Change in return on assets Bao et al. (2020); Dechow et al. (2011)  
CH_FCF Change in free cash flows Bao et al. (2020); Dechow et al. (2011)  
RE/ASSETS Retained earnings over total assets Bao et al. (2020); Summers and Sweeney (1998)  
EBIT/ASSETS Earnings before interest and taxes over total assets Bao et al. (2020); Summers and Sweeney (1998)  
ISSUE Actual issuance Bao et al. (2020); Dechow et al. (2011)  
B/M Book-to-market Bao et al. (2020); Dechow et al. (2011)  
Panel C: Snippet of the Dataset Compiled by Bao et al. (2020) 
fyeargvkeyp_aaermisstateCUR_ASSETSAPASSETSCEQC_ST_INVSCOGS⋯a% SOFT_ASSETSCH_CSH_SALESCH_CSH_MGCH_ROA
0 1990 1,009 NaN 0 10.047 3.736 32.335 6.262 0.002 30.633 ⋯ 0.312448 0.095082 0.082631 −0.019761 
1 1990 1,011 NaN 0 1.247 0.803 7.784 0.667 0.171 1.125 ⋯ 0.315904 0.188832 −0.211389 −0.117832 
2 1990 1,017 NaN 0 55.040 3.601 118.120 44.393 3.132 107.343 ⋯ 0.605342 0.097551 −0.105780 0.091206 
3 1990 1,021 NaN 0 24.684 3.948 34.591 7.751 0.411 31.214 ⋯ 0.793068 −0.005725 −0.249704 0.017545 
4 1990 1,028 NaN 0 17.325 3.520 27.542 −12.142 1.017 32.662 ⋯ 0.869182 −0.231536 −1.674893 −0.466667 
Panel C: Snippet of the Dataset Compiled by Bao et al. (2020) 
fyeargvkeyp_aaermisstateCUR_ASSETSAPASSETSCEQC_ST_INVSCOGS⋯a% SOFT_ASSETSCH_CSH_SALESCH_CSH_MGCH_ROA
0 1990 1,009 NaN 0 10.047 3.736 32.335 6.262 0.002 30.633 ⋯ 0.312448 0.095082 0.082631 −0.019761 
1 1990 1,011 NaN 0 1.247 0.803 7.784 0.667 0.171 1.125 ⋯ 0.315904 0.188832 −0.211389 −0.117832 
2 1990 1,017 NaN 0 55.040 3.601 118.120 44.393 3.132 107.343 ⋯ 0.605342 0.097551 −0.105780 0.091206 
3 1990 1,021 NaN 0 24.684 3.948 34.591 7.751 0.411 31.214 ⋯ 0.793068 −0.005725 −0.249704 0.017545 
4 1990 1,028 NaN 0 17.325 3.520 27.542 −12.142 1.017 32.662 ⋯ 0.869182 −0.231536 −1.674893 −0.466667 

See the sources for more detailed information on the variables used in this study. Following Bao et al. (2020), we used a set of 28 accounting and 14 financial ratio variables sourced from the dataset compiled by Bao et al. Panel C shows the snippet (first five records) of the dataset compiled by Bao et al. (2020). These variables, along with the dataset, were employed to develop the fraud detection model for this study. By applying XAI to the fraud detection model developed in this study, we demonstrate how the application of XAI provides valuable insights into model outputs and decisions, thereby benefiting stakeholders, such as auditors. This enhanced interpretability, understanding, and transparency can foster trust in AI applications and enable the auditing profession to unlock the full potential of AI technologies.

a The ellipses indicate that there are more columns, and the current table only shows a subset of data.

View Large

The confusion matrix3 allows us to evaluate the performance of the fraud detection classification model and displays the results in four categories: true positive (TP), true negative (TN), false positive (FP), and false negative (FN) (Figure 1). Using the values of TP, TN, FP, and FN, various performance metrics, such as accuracy,4 precision,5 and recall6 rates, were calculated to further evaluate the effectiveness of the model. The accuracy rate was computed as 78.76 percent by dividing the sum of TP and TN by the sum of TP, FP, TN, and FN. The precision rate was calculated as 0.757 by dividing the number of TP by the sum of TP and FP. The recall rate was calculated as 0.846 by dividing the number of TP by the sum of TP and FN.

FIGURE 1
The Confusion Matrix
View largeDownload slide

The Confusion Matrix

This figure shows the results of a confusion matrix, which is commonly used to evaluate the performance of a classification model, specifically, the fraud detection model in this study. The confusion matrix consisted of four main components: (1) true negative, representing the number of observations correctly predicted as negative (or nonfraudulent); (2) false positive, indicating the number of observations incorrectly predicted as positive (or fraudulent); (3) false negative, indicating the number of observations incorrectly predicted as negative (or nonfraudulent); and (4) true positive, representing the number of observations correctly predicted as positive (or fraudulent).

FIGURE 1
The Confusion Matrix
View largeDownload slide

The Confusion Matrix

This figure shows the results of a confusion matrix, which is commonly used to evaluate the performance of a classification model, specifically, the fraud detection model in this study. The confusion matrix consisted of four main components: (1) true negative, representing the number of observations correctly predicted as negative (or nonfraudulent); (2) false positive, indicating the number of observations incorrectly predicted as positive (or fraudulent); (3) false negative, indicating the number of observations incorrectly predicted as negative (or nonfraudulent); and (4) true positive, representing the number of observations correctly predicted as positive (or fraudulent).

Close modal

XAI encompasses a range of methods designed to increase the transparency of AI models by providing insights into their decision-making processes. The popular XAI methods demonstrated in this study include local interpretable model-agnostic explanations (LIME), Shapley additive explanations (SHAP), and counterfactual explanations. Whereas XAI methods are precise in evaluating the impact of individual variables, their evaluation is conducted within the context of a specific firm and its fiscal year, consistent with the standard practices of auditors, who typically audit a firm’s entire fiscal year.

In this study, LIME (Ribeiro, Singh, and Guestrin 2016), one of the most widely used XAI methods, was employed alongside other XAI methods to provide explanations for individual predictions made by the fraud detection model. LIME can provide insights into why the model made a specific prediction and identify the features that are most crucial in the model’s prediction.7 LIME accomplishes this by creating a local approximation of the model with a simpler one, determining the significant features, and expressing their contributions to the prediction using importance scores or weights. Using LIME, auditors can visualize the decision-making process of the model and gain insights into the rationale behind its predictions, given that interpretability is becoming an important prerequisite for AI models (see Figure 2). Interpreting model predictions at this deep level would not have been possible without the aid of XAI, which is particularly useful for stakeholders who are not involved in the development of AI models.

FIGURE 2
LIME Explanation of Prediction Probabilities for a Single “Fraud” Observation of Enron for Year 2000
View largeDownload slide

LIME Explanation of Prediction Probabilities for a Single “Fraud” Observation of Enron for Year 2000

This figure shows the prediction probabilities for a single fraud observation, indicating a 0.96 probability of classifying it as fraudulent and a 0.04 probability of classifying it as nonfraudulent. The bar graph in the center highlights the features that contribute the most to the prediction outcomes for the fraud and nonfraud classes. The length of each horizontal bar represents the LIME weight for each feature, indicating the impact of each feature on the prediction outcome rather than the direct probability. Evidently, the feature property, plant, and equipment (PPE) has the greatest influence on the prediction outcome of nonfraud, as its LIME weight is highest (0.07), whereas the feature current assets (CUR_ASSETS) has the greatest influence on the prediction outcome of fraud, as its LIME weight is highest (0.04). The features highlighted in orange contribute to fraud prediction, whereas those highlighted in blue contribute to nonfraud prediction. The table on the far right displays the top features and their corresponding actual values that contributed to the predictions.

FIGURE 2
LIME Explanation of Prediction Probabilities for a Single “Fraud” Observation of Enron for Year 2000
View largeDownload slide

LIME Explanation of Prediction Probabilities for a Single “Fraud” Observation of Enron for Year 2000

This figure shows the prediction probabilities for a single fraud observation, indicating a 0.96 probability of classifying it as fraudulent and a 0.04 probability of classifying it as nonfraudulent. The bar graph in the center highlights the features that contribute the most to the prediction outcomes for the fraud and nonfraud classes. The length of each horizontal bar represents the LIME weight for each feature, indicating the impact of each feature on the prediction outcome rather than the direct probability. Evidently, the feature property, plant, and equipment (PPE) has the greatest influence on the prediction outcome of nonfraud, as its LIME weight is highest (0.07), whereas the feature current assets (CUR_ASSETS) has the greatest influence on the prediction outcome of fraud, as its LIME weight is highest (0.04). The features highlighted in orange contribute to fraud prediction, whereas those highlighted in blue contribute to nonfraud prediction. The table on the far right displays the top features and their corresponding actual values that contributed to the predictions.

Close modal

Next, we delve into another XAI method, SHAP (Lundberg and Lee 2017), which can make the complex decision-making process of AI models more transparent by breaking down the model’s prediction. SHAP achieves this by assigning an “importance” score to each feature that influences the model’s decision. Furthermore, SHAP provides a range of visualization tools to enhance the interpretability of the model. By examining the ranking of features, end-users can visually assess the extent to which each feature contributes to the model’s decision. Figure 3 shows the SHAP force plot and illustrates the identical fraud observation of Enron for 2000, which we previously discussed using LIME. Using SHAP’s force plot, auditors can identify the key features that contribute to moving the model’s prediction away from the base value toward the final prediction. The interpretability of individual predictions provides a more intuitive understanding than the overall interpretability of the model as it captures useful details that might be missed at a broader level.

FIGURE 3
A Force Plot That Explains How Key Features Drive Up or Down the Probability for Prediction of a Single “Fraud” Observation of Enron for Year 2000
View largeDownload slide

A Force Plot That Explains How Key Features Drive Up or Down the Probability for Prediction of a Single “Fraud” Observation of Enron for Year 2000

This figure shows a SHAP force plot, a visual representation of feature importance, which explains how features in the red segments increase the probability of fraud prediction, whereas the features in the blue segments decrease the probability of fraud prediction. The bold value, 0.96, is the actual predicted probability for this specific observation that was predicted as fraudulent, and the base value is 0.5, representing the average predicted probability across the dataset, which remains constant for all observations. The features are ordered by segment size and arranged in descending order based on their impact on the prediction. The features in the red segments that have a positive impact on fraud prediction constitute current liabilities (CUR_LIAB), sale of common and preferred stock (SALE_CS_PS), accounts payable (AP), change in cash sales (CH_CSH_SALES), soft assets (% SOFT_ASSETS), and receivables (REC), whereas the features in the blue segments that have a negative impact on the fraud prediction constitute property, plant, and equipment (PPE). The actual values of these features in the Enron observation are indicated below the red and blue segments.

FIGURE 3
A Force Plot That Explains How Key Features Drive Up or Down the Probability for Prediction of a Single “Fraud” Observation of Enron for Year 2000
View largeDownload slide

A Force Plot That Explains How Key Features Drive Up or Down the Probability for Prediction of a Single “Fraud” Observation of Enron for Year 2000

This figure shows a SHAP force plot, a visual representation of feature importance, which explains how features in the red segments increase the probability of fraud prediction, whereas the features in the blue segments decrease the probability of fraud prediction. The bold value, 0.96, is the actual predicted probability for this specific observation that was predicted as fraudulent, and the base value is 0.5, representing the average predicted probability across the dataset, which remains constant for all observations. The features are ordered by segment size and arranged in descending order based on their impact on the prediction. The features in the red segments that have a positive impact on fraud prediction constitute current liabilities (CUR_LIAB), sale of common and preferred stock (SALE_CS_PS), accounts payable (AP), change in cash sales (CH_CSH_SALES), soft assets (% SOFT_ASSETS), and receivables (REC), whereas the features in the blue segments that have a negative impact on the fraud prediction constitute property, plant, and equipment (PPE). The actual values of these features in the Enron observation are indicated below the red and blue segments.

Close modal

Next, we employed SHAP’s decision plot to offer insights into the model’s behavior and visualize how model predictions change across different feature values. Figure 4 presents two decision plots: one for the identical fraud observation of Enron from 2000, as previously discussed, and another for a nonfraud observation of Apple from the same year. Using SHAP’s decision plot, auditors can gain additional insights by visualizing how each feature influences the movement of the prediction line, indicating whether a feature contributes positively or negatively to the prediction outcome. This information can help understand which features are essential for making predictions and the underlying reasons behind model predictions, thereby demystifying the model’s decision-making process.

FIGURE 4

Decision Plots That Illustrate How Models Make Predictions

FIGURE 4

Decision Plots That Illustrate How Models Make Predictions

Panel A:
Decision Plot of a Single “Fraud” Observation of Enron for Year 2000
View largeDownload slide

Decision Plot of a Single “Fraud” Observation of Enron for Year 2000

Panel A:
Decision Plot of a Single “Fraud” Observation of Enron for Year 2000
View largeDownload slide

Decision Plot of a Single “Fraud” Observation of Enron for Year 2000

Close modal
Panel B:
Decision Plot of a Single “Nonfraud” Observation of Apple for Year 2000
View largeDownload slide

Decision Plot of a Single “Nonfraud” Observation of Apple for Year 2000

This figure shows SHAP decision plots (i.e., how models make decisions), where the y-axis represents the features, the x-axis represents the SHAP values, which indicate the impact of a feature on the model’s output, the vertical line represents the base value, and the colored line represents the prediction line. The actual values of the features in the given observation are displayed next to the prediction line. The point at which the prediction line intersects the x-axis at the top represents the predicted probability, which ranges from 0 to 1. If the probability is above 0.5, the observation is classified as fraudulent; if it is below 0.5, the observation is classified as nonfraudulent. The features are arranged in descending order based on the magnitude of their impacts, with the most influential features positioned at the top. As can be observed in Panel A, the feature current liabilities (CUR_LIAB) has the largest positive impact on the probability of fraud prediction, causing the prediction line to shift toward the right (closer to 1). On the other hand, in Panel B, the feature soft assets (% SOFT_ASSETS) has the largest positive impact on the probability of nonfraud prediction, causing the prediction line to shift toward the left (closer to 0). By examining the features from the bottom to the top, we can gain insights into the direction of influence each feature has on the prediction line, shifting it toward either the right (closer to 1) or the left (closer to 0).

Panel B:
Decision Plot of a Single “Nonfraud” Observation of Apple for Year 2000
View largeDownload slide

Decision Plot of a Single “Nonfraud” Observation of Apple for Year 2000

This figure shows SHAP decision plots (i.e., how models make decisions), where the y-axis represents the features, the x-axis represents the SHAP values, which indicate the impact of a feature on the model’s output, the vertical line represents the base value, and the colored line represents the prediction line. The actual values of the features in the given observation are displayed next to the prediction line. The point at which the prediction line intersects the x-axis at the top represents the predicted probability, which ranges from 0 to 1. If the probability is above 0.5, the observation is classified as fraudulent; if it is below 0.5, the observation is classified as nonfraudulent. The features are arranged in descending order based on the magnitude of their impacts, with the most influential features positioned at the top. As can be observed in Panel A, the feature current liabilities (CUR_LIAB) has the largest positive impact on the probability of fraud prediction, causing the prediction line to shift toward the right (closer to 1). On the other hand, in Panel B, the feature soft assets (% SOFT_ASSETS) has the largest positive impact on the probability of nonfraud prediction, causing the prediction line to shift toward the left (closer to 0). By examining the features from the bottom to the top, we can gain insights into the direction of influence each feature has on the prediction line, shifting it toward either the right (closer to 1) or the left (closer to 0).

Close modal

Unlike the previously discussed XAI methods (LIME, SHAP’s force plot, SHAP’s decision plot) that focus on explaining individual predictions, SHAP’s summary plot provides a global understanding of which features have a significant impact on model predictions (see Figure 5). Using the SHAP summary plot, auditors can make informed decisions regarding feature selection and improve model performance by selecting the most influential features to build AI models.

FIGURE 5

Summary Plots That Highlight the Selected Important Features and Their Impact on “Fraud” and “Nonfraud” Predictions

FIGURE 5

Summary Plots That Highlight the Selected Important Features and Their Impact on “Fraud” and “Nonfraud” Predictions

Panel A:
Fraud Observation
View largeDownload slide

Fraud Observation

Panel A:
Fraud Observation
View largeDownload slide

Fraud Observation

Close modal
Panel B:
Nonfraud Observation
View largeDownload slide

Nonfraud Observation

This figure displays SHAP summary plots that provide a comprehensive overview of the SHAP values for the selected important features across the entire dataset and how these features impact fraud (Panel A) and nonfraud (Panel B) predictions. The y-axis represents the features, and the x-axis represents the SHAP values, which indicate the impact of each feature on the prediction of fraud or nonfraud. Each point on the plot corresponds to a specific feature and its SHAP value, and the red color represents high values of features, whereas the blue color represents low values of features. The high and low values for each feature represent the range of values observed in the dataset for that particular feature. The features are ordered based on their importance, with the most important features placed at the top. Feature ranking was determined by considering the collective impact of each feature across all instances in the dataset. For example, in Panel A, high values of the feature current assets (CUR_ASSETS), indicated in red, increase the probability of being predicated as fraudulent, whereas low values of the same feature, indicated in blue, decrease the probability of being predicted as fraudulent. Conversely, in Panel B, low values of the feature current assets (CUR_ASSETS), indicated in blue, increase the probability of being predicted as nonfraudulent, whereas high values of the same feature, indicated in red, decrease the probability of being predicted as nonfraudulent. Similarly, in Panel A, low values of the feature property, plant, and equipment (PPE), indicated in blue, increase the probability of being predicated as fraudulent, whereas high values of the same feature, indicated in red, decrease the probability of being predicted as fraudulent. Conversely, in Panel B, high values of the feature property, plant, and equipment (PPE), indicated in red, increase the probability of being predicted as nonfraudulent, whereas low values of the same feature, indicated in blue, decrease the probability of being predicted as nonfraudulent.

Panel B:
Nonfraud Observation
View largeDownload slide

Nonfraud Observation

This figure displays SHAP summary plots that provide a comprehensive overview of the SHAP values for the selected important features across the entire dataset and how these features impact fraud (Panel A) and nonfraud (Panel B) predictions. The y-axis represents the features, and the x-axis represents the SHAP values, which indicate the impact of each feature on the prediction of fraud or nonfraud. Each point on the plot corresponds to a specific feature and its SHAP value, and the red color represents high values of features, whereas the blue color represents low values of features. The high and low values for each feature represent the range of values observed in the dataset for that particular feature. The features are ordered based on their importance, with the most important features placed at the top. Feature ranking was determined by considering the collective impact of each feature across all instances in the dataset. For example, in Panel A, high values of the feature current assets (CUR_ASSETS), indicated in red, increase the probability of being predicated as fraudulent, whereas low values of the same feature, indicated in blue, decrease the probability of being predicted as fraudulent. Conversely, in Panel B, low values of the feature current assets (CUR_ASSETS), indicated in blue, increase the probability of being predicted as nonfraudulent, whereas high values of the same feature, indicated in red, decrease the probability of being predicted as nonfraudulent. Similarly, in Panel A, low values of the feature property, plant, and equipment (PPE), indicated in blue, increase the probability of being predicated as fraudulent, whereas high values of the same feature, indicated in red, decrease the probability of being predicted as fraudulent. Conversely, in Panel B, high values of the feature property, plant, and equipment (PPE), indicated in red, increase the probability of being predicted as nonfraudulent, whereas low values of the same feature, indicated in blue, decrease the probability of being predicted as nonfraudulent.

Close modal

Counterfactual explanations generate hypothetical scenarios that would change the outcome, for instance, from fraud to nonfraud or vice versa, by making small changes to the input features. These explanations answer questions such as “What would need to change in the input for the prediction to be different?” These “what-if” scenarios are particularly useful for stakeholders interested in understanding how they can influence the predicted outcome, such as avoiding being classified as fraudulent. Table 2 presents the potential impact of modifying specific feature values on the prediction of the model, shifting it from fraud to nonfraud. Ultimately, stakeholders must determine which illustrative scenarios seem plausible.

TABLE 2

Counterfactual Explanations for a Snippet of a Data Observation Predicted as “Fraud,” Which Was Originally a Fraudulent Observation of Enron for Year 2000

PredictionCUR_ASSETSAPRECRECH_CSH_SALESCur_LIAB% SOFT_ASSETS…a
Original Observation Fraud 3,0381.0 9,777.0 1,2270.0 2,030.0 1.3 28,406.0 0.8 … 
Modified Observation 1 Nonfraud 3,0381.0 9,777.0 57,414.9 364,352.2 1.3 28,406.0 0.8 … 
Modified Observation 2 Nonfraud 3,0381.0 15,979.9 12,270.0 2,030.0 1.3 28,406.0 0.8  
Modified Observation 3 Nonfraud 3,0381.0 27,937.4 12,270.0 2,030.0 1.3 113,143.1 0.8  
PredictionCUR_ASSETSAPRECRECH_CSH_SALESCur_LIAB% SOFT_ASSETS…a
Original Observation Fraud 3,0381.0 9,777.0 1,2270.0 2,030.0 1.3 28,406.0 0.8 … 
Modified Observation 1 Nonfraud 3,0381.0 9,777.0 57,414.9 364,352.2 1.3 28,406.0 0.8 … 
Modified Observation 2 Nonfraud 3,0381.0 15,979.9 12,270.0 2,030.0 1.3 28,406.0 0.8  
Modified Observation 3 Nonfraud 3,0381.0 27,937.4 12,270.0 2,030.0 1.3 113,143.1 0.8  

This table illustrates counterfactual explanations, where the first row represents a sample input data point that was originally a fraud observation and was predicted as “fraud.” The second, third, and fourth rows present counterfactual explanations in which the feature values from the original data point were perturbed. For example, it suggests that if the value of the receivables (REC) feature were changed from 12,270.0 to 57,414.9 and the value of retained earnings (RE) were changed from 2,030.0 to 364,352.2, the observation would be classified as a “nonfraud” case, as shown in the second row. In the third row, for Modified Observation 2, it suggests that when accounts payable (AP) were changed from 9,777.0 to 15,979.9, while keeping other features constant, the prediction shifts to “nonfraud.” In the fourth row, for Modified Observation 3, it suggests that when accounts payable (AP) changed from 9,777.0 to 27,937.4 and current liabilities (CUR_LIAB) were changed from 28,406.0 to 113,143.1, the prediction shifted to “nonfraud.” Ultimately, it is up to stakeholders to evaluate the plausibility of these illustrative scenarios, as some of them are unlikely to be true.

a The ellipses indicate that there are more columns, and the current table only shows a subset of data.

View Large

Although it is important to construct interpretable and explainable AI models, it is also imperative to develop responsible and trustworthy AI applications. As AI applications continue to become mainstream and expand in scope and complexity, regulatory measures are necessary to ensure that AI systems do not perpetuate or exacerbate existing biases or discriminate against protected groups. Overall, the objective of AI regulation is to strike a balance between promoting innovation and ensuring the responsible use of AI technologies for individuals, society, and the economy (Miller 2019).

As AI applications transcend national boundaries, collaborative initiatives in the international space can help to establish consistent standards and guidelines for the responsible use of AI across countries. Figure 6 provides a brief overview of AI initiatives undertaken by different countries in their efforts to regulate AI. As the regulatory landscape for AI evolves, policymakers must regularly assess the effectiveness of existing regulations and adapt them to keep pace with the technological advancements and emerging challenges. Using XAI, this study offers insights for policymakers to address emerging challenges related to balancing AI’s advanced capabilities with the industry’s demands for transparency and responsibility.

FIGURE 6
Emerging Global AI Regulations
View largeDownload slide

Emerging Global AI Regulations

This figure summarizes recent developments in AI regulation, which aim to address ethical concerns, ensure transparency and accountability, and manage the potential risks associated with AI technologies.

FIGURE 6
Emerging Global AI Regulations
View largeDownload slide

Emerging Global AI Regulations

This figure summarizes recent developments in AI regulation, which aim to address ethical concerns, ensure transparency and accountability, and manage the potential risks associated with AI technologies.

Close modal
Adadi
,
A.
, and
M.
Berrada
.
2018
.
Peeking inside the black-box: A survey on explainable artificial intelligence (XAI)
.
IEEE Access
6
:
52138
–
52160
. https://doi.org/10.1109/ACCESS.2018.2870052
Google Scholar
Crossref
Search ADS
 
Bao,
Y.
,
B.
Ke
,
B.
Li
,
Y. J.
Yu
, and
J.
Zhang
.
2020
.
Detecting accounting fraud in publicly traded U.S. firms using a machine learning approach
.
Journal of Accounting Research
58
(
1
):
199
–
235
.
https://doi.org/10.1111/1475-679X.12292
Google Scholar
Crossref
Search ADS
 
Beneish
,
M. D.
1999
.
The detection of earnings manipulation
.
Financial Analysts Journal
55
(
5
):
24
–
36
. https://doi.org/10.2469/faj.v55.n5.2296
Google Scholar
Crossref
Search ADS
 
Breiman,
L.
2001
.
Random forests
.
Machine Learning
45
(
1
):
5
–
32
.
https://doi.org/10.1023/A:1010933404324
Google Scholar
Crossref
Search ADS
 
Cecchini,
M.
,
H.
Aytug
,
G. J.
Koehler
, and
P.
Pathak
.
2010
.
Detecting management fraud in public companies
.
Management Science
56
(
7
):
1146
–
1160
.
https://doi.org/10.1287/mnsc.1100.1174
Google Scholar
Crossref
Search ADS
 
Dechow,
P. M.
,
W.
Ge
,
C. R.
Larson
, and
R. G.
Sloan
.
2011
.
Predicting material accounting misstatements
.
Contemporary Accounting Research
28
(
1
):
17
–
82
.
https://doi.org/10.1111/j.1911-3846.2010.01041.x
Google Scholar
Crossref
Search ADS
 
Došilović,
F. K.
,
M.
Brčić
, and
N.
Hlupić
.
2018
.
Explainable artificial intelligence: A survey
.
Proceedings of the 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)
,
Opatija, Croatia
,
May 21–25
.
http://dx.doi.org/10.23919/MIPRO.2018.8400040
Google Scholar
 
Islam,
S. R.
,
W.
Eberle
,
S. K.
Ghafoor
, and
M.
Ahmed
.
2021
.
Explainable artificial intelligence approaches: A survey
. (Working paper). https://arxiv.org/abs/2101.09429
Google Scholar
 
Lipton,
Z. C.
2018
.
The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery
.
Queue
16
(
3
):
31
–
57
.
https://doi.org/10.1145/3236386.3241340
Google Scholar
Crossref
Search ADS
 
Lundberg,
S. M.
, and
S.
Lee
.
2017
.
A unified approach to interpreting model predictions
.
NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems
.
Long Beach, CA
,
December 4–9
.
Google Scholar
 
Miller,
T.
2019
.
Explanation in artificial intelligence: Insights from the social sciences
.
Artificial Intelligence
267
:
1
–
38
.
https://doi.org/10.1016/j.artint.2018.07.007
Google Scholar
Crossref
Search ADS
 
PwC
.
2022
.
Sizing the prize What’s the real value of AI for your business and how can you capitalise?
https://www.pwc.com/gx/en/issues/analytics/assets/pwc-ai-analysis-sizing-the-prize-report.pdf
Ribeiro,
M. T.
,
S.
Singh
, and
C.
Guestrin
.
2016
.
“Why should I trust you?” Explaining the predictions of any classifier
.
Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
.
San Francisco, CA
,
August 13–17
.
Google Scholar
 
Summers,
S. L.
, and
J. T.
Sweeney
.
1998
.
Fraudulently misstated financial statements and insider trading: An empirical analysis
.
Accounting Review
73
:
131
–
146
.
Google Scholar
 
Turek,
M.
2018
.
Explainable artificial intelligence (XAI)
. Defense Advanced Research Projects Agency. http://web.archive.org/web/20190728055815/https://www.darpa.mil/program/explainable-artificial-intelligence
Google Scholar
 
Zhang,
C. A.
,
S.
Cho
, and
M.
Vasarhelyi
.
2022
.
Explainable artificial intelligence (XAI) in auditing
.
International Journal of Accounting Information Systems
46
:
100572
.
https://doi.org/10.1016/j.accinf.2022.100572
Google Scholar
Crossref
Search ADS
 
1

The ensemble learning algorithm is known for its effectiveness in handling complex tasks and improving accuracy.

2

The random forest method is an ensemble learning algorithm widely used for classification tasks. It combines multiple decision trees to create a forest of trees, where each tree makes its own predictions. In a random forest, each tree is trained on a random subset of the training data and a random subset of features. This randomness helps in reducing overfitting of the model and improving the robustness of the model. In a random forest, the final prediction is determined by aggregating the predictions of all individual trees, usually through majority voting. Random forests are known for delivering reliable predictions for high dimensional datasets (Breiman 2001).

3

Confusion matrix is commonly used to evaluate the performance of a classification model.

4

Accuracy represents the proportion of total observations that were correctly classified by the model.

5

Precision indicates the proportion of the predicted fraud cases that were correctly classified by the model.

6

Recall indicates the proportion of the actual fraud cases that were correctly classified by the model.

7

The term “feature” is often used interchangeably with “variable” in the context of classification models. Therefore, in this study, we refer to variables as features, and both terms are used interchangeably throughout.

2024