SUMMARY
To minimize the associated risks (e.g., litigation, regulatory scrutiny) of implementing increasingly advanced data and analytics-based substantive auditing techniques, audit firms should ensure that key audit stakeholder groups sufficiently understand such procedures and believe that they maintain or elevate audit quality. However, little is known about how various stakeholder groups view data and analytics-based substantive procedures. Ballou, Grenier, and Reffett (2021) address this question by examining how three key audit stakeholder groups (investors, jurors, and AICPA peer reviewers) view two commonly employed data and analytics-based auditing techniques (population testing and predictive modeling). Our paper summarizes Ballou et al.'s (2021) study by summarizing its research questions, experimental method, and results. We then conclude with a discussion of the study's implications for audit practice and, in particular, the steps that audit firms should take to ensure stakeholder comfort.
I. INTRODUCTION
Technological advances have greatly expanded data availability, particularly from non-client sources, and have also enhanced audit firms' processing and analyzing capabilities. Such advancements have enabled audit firms to develop innovative data and analytics-based audit techniques that are increasingly used for substantive testing purposes (Cao, Chychyla, and Stewart 2015; Earley 2015; Appelbaum, Kogan, and Vasarhelyi 2017). The use of data and analytics-based substantive procedures, particularly those that are predictive in nature, could transform audit practice through enhanced insights to the auditors and client management, while also improving audit efficiency.
Two common data and analytics-based techniques are population testing and predictive analytical procedures using models (hereafter, predictive modeling). Population testing is used to evaluate whole populations of accounts and transactions in order to discover items that have certain anomalous characteristics that are indicative of potential misstatement, and thus require further investigation. Predictive modeling involves the use of historical data to identify factors that are predictive of particular account balances, and using those factors to develop and calibrate a regression model to generate a predicted current year balance. Potentially used as a substantive analytical procedure, model predictions are then compared to amounts reported in the unaudited financial statements and discrepancies are investigated.1 Firms have invested significant resources to advance their technological capabilities including technologies that ensure advanced data, and analytics-based audit procedures improve both audit effectiveness and audit efficiency (Forbes Insights 2018; Bloomberg Tax 2020; Tysiac 2020). However, a potentially overlooked, but critical, step to minimize the associated risks (e.g., litigation, regulatory scrutiny) of implementing them is ensuring that key audit stakeholders are confident that these newly developed audit procedures improve, or at least maintain, audit effectiveness. Failing to do so would greatly increase audit firms' exposure to various risks, particularly regulatory and litigation risk (Ballou et al. 2021).
Until recently, little was known about how various audit stakeholder groups view data and analytics-based auditing procedures. Thus, the extent to which audit firms need to convince different stakeholder groups of the merits of data and analytics-based audit procedures was unclear. Ballou et al. (2021) address this question by examining how three key audit stakeholder groups—investors, jurors, and AICPA peer reviewers—view two emerging data and analytics-based auditing techniques—population testing and predictive modeling—versus traditional statistical sample testing of an account population.2 More specifically, Ballou et al. (2021) investigate the following general research questions:
Is investor willingness to invest in a company affected by whether the company's auditors used statistical sampling, population testing, or predictive modeling procedures, and if so, is it contingent on the risk of material misstatement?
In cases of undetected fraud, are jurors' auditor negligence verdicts affected by whether the auditors used statistical sampling, population testing, or predictive modeling procedures?
Do AICPA peer reviewers view data and analytics-based procedures (population testing or predictive modeling) more or less favorably than traditional statistical sampling, and do their perceptions depend on the risk of misstatement?
II. METHOD AND RESULTS
Ballou et al. (2021) investigate their research questions by conducting three experiments, one for each of the three aforementioned stakeholder groups. A summary of research questions, participants, key findings, and implications can be found in Table 1. Overall, the three experiments yielded interesting results that generally, but not entirely, indicate that the examined stakeholders are comfortable with auditors' use of population testing and predictive modeling audit techniques as a means of obtaining substantive audit evidence.
Experiment 1: Non-Professional Investors
Non-professional investor participants were recruited from Amazon Mechanical Turk, an online labor market that pays individuals to complete human intelligence tasks. The sample included 232 individuals, of which 61 percent were male, average age was 35, and average investing experience was self-reported as moderate.
Non-professional investor participants read background information about financial statements and the audit process before assuming the role of an investor who is considering investing in an insurance company. Participants were provided background information including a comparative balance sheet and income statement, an audit report, and information about the actual procedures performed. After reading the case, participants then indicated their willingness to invest and provided their perceptions of auditor competence and the justifiability of the auditors' procedures. The experiment manipulated two independent variables across participants: (1) risk of material misstatement (RMM) as high or low; and (2) the nature of the auditors' procedures as statistical sampling, population testing, or predictive modeling.3
The results indicate that the nature of the auditors' procedures had no effect on investor participants' intention to invest. However, investors evaluated predictive modeling more favorably than statistical sampling procedures for several measures. In particular, when RMM is high, investor participants perceived auditors as more competent and their procedures as more justified when the auditors used predictive modeling rather than statistical sampling approaches. Overall, these findings imply that non-professional investors view new data and analytics-based audit techniques as positively, if not more so under certain conditions, as they view more traditional statistical sampling procedures.
Experiment 2: Jurors
Juror participants also were recruited from Amazon Mechanical Turk. The sample included 201 participants from Amazon Mechanical Turk who confirmed that they were U.S. citizens and at least 18 years old, making them legally eligible to be a juror. Of the 201 participants, 48 percent were male and the average age was 36 years. Juror participants were given background information about financial statements and the audit process before assuming the role of a juror in a fictitious lawsuit between an insurance company and its auditor. The case included background information about the insurance company, a summary of the audit procedures performed, and an undetected $5 million material misstatement in the claims account (and subsequently net income). After reading the case, participants provided their perceptions of auditor competence, justifiability of the auditors' procedures, and whether the auditors were (or were not) negligent in their performance of the audit. The experiment manipulated the auditors' procedures between statistical sampling, population testing, and predictive modeling procedures.
Results indicate that juror participants' judgments of auditor negligence were not significantly different across the population testing versus statistical sampling conditions. Similarly, juror participants' judgments of auditor negligence were not significantly different in the predictive modeling versus statistical sampling conditions. However, juror participants were less likely (at a marginal level of statistical significance) to indicate that the auditors were negligent in the population testing versus predictive modeling condition.
To help better understand the difference in jurors' negligence judgments in the population testing versus predictive modeling conditions, Ballou et al. (2021) also examined other juror judgments. Participants in the population testing condition perceived the auditors' processes to be more justified, and exhibited fewer negative emotions toward the auditors, than those in the predictive modeling condition. Furthermore, there are strong correlations between jurors' auditor negligence judgments and their perceptions of both the defensibility of the auditors' methods and their emotional reactions to the auditors.
Experiment 3: AICPA Peer Reviewers
AICPA peer reviewer participants were recruited via email. The authors sent 1,857 emails (excluding 169 undeliverable emails) to peer reviewers listed on the AICPA website, which yielded a sample of 88 AICPA peer reviewers. Participants had an average 27 years of experience in public accounting and 13 years of experience as AICPA peer reviewers. The majority were from small local or regional accounting firms (66 participants), and the remaining were from large regional firms (18 participants) or national firms (4 participants).4
AICPA peer reviewer participants read a case about an audit of an insurance company in their role as peer reviewers. The case provided information about the auditors' procedures to test the insurance company's claims expense and an audit adjustment that was made as a result of those procedures. After reading the case, peer reviewer participants rated the quality of audit procedures on a scale of 0 to 100. They also gave feedback on their familiarity with the procedures and provided their opinion (and justification) as to the most appropriate audit procedures for the scenario. Similar to the investor study, this study manipulated two independent variables: (1) RMM; and (2) the nature of the auditors' procedures. RMM was manipulated between high and low for the claims account. The auditors' procedures were manipulated between statistical sampling, population testing, and predictive modeling.
Results for AICPA peer reviewers indicate no significant differences in their quality ratings across statistical sampling, population testing, and predictive modeling procedures. While there was a strong positive association between firm size and AICPA peer reviewers' expertise with population testing procedures, there was not a significant correlation between firm size and AICPA peer reviewers' familiarity with predictive modeling. Similarly, there were not significant correlations between firm size and AICPA peer reviewers' perceptions of the quality of auditors' population testing procedures or predictive modeling procedures.
About one-third of AICPA peer reviewers who provided an open-ended response expressed concern about the dependability of data utilized in predictive modeling and/or its suitability as a primary substantive test. These AICPA peer reviewers seem to view predictive modeling as a supplement to, rather than a replacement for, more traditional substantive procedures. In conclusion, AICPA peer reviewers seem to be open to data and analytic auditing approaches in general, but have concerns about predictive modeling as a primary substantive test.
III. IMPLICATIONS FOR PRACTICE AND ACADEMIA
The results summarized in this article offer preliminary evidence for public accounting firms to consider as they innovate audit methodologies and procedures to leverage the many technological advancements transforming business.5Ballou et al. (2021) provide preliminary evidence that stakeholders do not appear to have significant concerns about data and analytics-based audit procedures with one exception. Both jurors and AICPA peer reviewers appear to have concerns about using predictive modeling as a primary source of substantive evidence.
Juror views are important because, even with enhanced audit effectiveness, data and analytics-based auditing procedures do not provide absolute assurance. In other words, undetected material misstatements will persist although perhaps at a reduced rate. Thus, it is imperative that firms understand how jurors and other triers of fact (e.g., judges) will evaluate their use of data and analytics in cases of alleged audit failure. Recent studies find elevated liability when using technology (e.g., data and analytics) in substantive testing under certain conditions (cf. Backof, Grenier, and Rasso 2021), but also identify effective defense strategies to mitigate elevated liability associated with technology (Grenier, Holman, Lowe, and Ulla 2021). Thus, in developing strategies to persuade jurors (and potentially regulators) of the effectiveness of newly developed audit procedures, particularly predictive modeling procedures, audit firms should consider the growing body of academic evidence.
Additional research on the stakeholder groups represented in this study, along with the addition of more sophisticated investors (e.g., institutional investors) and public accounting regulators for auditing firms (e.g., PCAOB in the United States), is critical for ensuring that all stakeholders move forward in a coordinated manner. For example, the results of Ballou et al. (2021), along with the ongoing research highlighted above, would inform the PCAOB's (2022) Data and Technology Task Force's efforts to ensure that PCAOB standards and guidance are responsive to firms' changing technologies.
While academics likely will generate such research in the coming years, audit firms might consider partnering with experienced researchers to gather systematic evidence from their firms' client stakeholders, including audit committees, senior management, internal auditors, and controller-area personnel. Such research collaborations should build on the initial findings reported within Ballou et al. (2021) but also be tailored to individual firms' specific technologies. For example, Deloitte (2021) utilizes its Omnia audit platform, EY (2021) its Helix platform, KPMG (2021) its Clara platform, and PwC (2021) its Aura platform, and all mention that data and analytics tools and technologies are transforming the audit. Further, these and other firms are investing in intelligent automation, including such technologies as data interrogation, robotic-process automation, and artificial intelligence that are expected to further increase both audit effectiveness and efficiency going forward (cf. Analytics India 2020).6 Further, technology is changing how companies process, record, and manage data. Advances such as those mentioned and blockchain are leading audit firms to consider alternatives to traditional audit testing techniques. EY notes in its white paper on the impacts of blockchain that the technology has the ability to use plug-in analyzers for testing populations of blockchain transactions to perform real-time audits of complete transactions (Percy 2016).
REFERENCES
Analytical procedures are used for planning (e.g., risk assessment/attention-directing) and final review phases of the audit; however, they also can be used for substantive (i.e., testing account balances) purposes. Ballou et al. (2021) examine predictive modeling used for substantive testing purposes. Substantive analytical procedures vary in their use of data and technology. In fact, firms have used regression models to generate expected balances as part of their substantive analytical procedures for many years (cf. Wilson and Weld 1997). What has more recently changed is that several technological advances such as expanded processing capacity along with increased data availability have enhanced the precision of auditors' models to an extent that enables their more widespread use for substantive procedures (cf. Strait and Estes 2021).
The study examines the selected stakeholder groups (i.e., investors, jurors, and AICPA peer reviewers), because auditors are potentially accountable to each group. With respect to investors, while they (i.e., investors) generally are not informed of the specific nature of the auditors' procedures, there are several important exceptions. First, investors could learn of the specific nature of auditors' procedures in the event of litigation. Specifically, in the event of bankruptcy, investors often evaluate the merits of pursuing legal claim(s) against the auditors. The probability that investors pursue such litigation depends on their (and their attorneys') perceptions of the merits of a potential claim against the auditors, which in turn are impacted by their perceptions of the quality of the auditors' substantive procedures (Maksymov et al. 2020). Second, investors potentially could learn of the specific nature of the auditors' procedures through the auditors' Critical Audit Matter (CAM) disclosure(s).
An experimental manipulation is where randomly assigned groups of participants see different information than other groups. This allows researchers to attribute any differences in judgments across the two groups to the manipulated informational differences. For example, the experiment manipulates the assessed level of RMM for the insurance client engagement at two levels, high and low. In the high RMM condition, the auditor considers insurance claims expense to be a high-risk area because historical predictions of the insurance client's wildfire claims (a major component of its claims expense) have been relatively imprecise. The instrument further indicates that the auditor disclosed a critical audit matter (CAM) relating to claims expense. In the low RMM condition, the auditor considers claims expense to be a low risk area as historical reductions have been relatively precise, but that given its significance to the financial statements, chose to disclose a CAM relating to claims expense.
Note that the use of AICPA peer reviewers presents a limitation to the study. The composition of this population suggests that relatively few of these practitioners would have experience in performing or conducting peer reviews involving audits of public companies. While PCAOB reviewers would be more representative of reviewers of public companies, Ballou et al. (2021) were unable to get the PCAOB to participate in the study.
Ballou et al. (2021) only examine aggregate stakeholder evaluations of the different types of auditing procedures, and do not assess whether the stakeholder actually understands the nature of the auditing procedures and the associated level of assurance provided. As such, the observed effects (or lack thereof) could be attributable to stakeholder misunderstanding of the nature of the audit procedures. This possibility represents a limitation of the study, and an opportunity for future research to examine stakeholder understanding of different audit procedures.
For example, KPMG describes how an engagement team used robotic process automation to aid the team in reconciling data, selecting samples, preparing client requests, and other detailed tasks to improve audit efficiency with enhanced quality and reduced disruption—freeing up associates to focus on higher risk areas (KPMG 2022).
