SUMMARY
On June 26, 2023, the Public Company Accounting Oversight Board (the Board or PCAOB) issued a request for comment on its Proposed Amendments Related to Aspects of Designing and Performing Audit Procedures That Involve Technology-Assisted Analysis of Information in Electronic Form (Public Company Accounting Oversight Board (PCAOB) 2023a). This commentary summarizes the participating committee members’ views on the proposal. We first provide answers to specific questions posed in the release, viewing the issuance of a new standard as a given. Subsequently, we also examine how well the proposal’s economic analysis establishes a solid foundation for new standard setting.
I. INTRODUCTION
We are pleased to provide feedback on the PCAOB’s Proposed Amendments Related to Aspects of Designing and Performing Audit Procedures That Involve Technology-Assisted Analysis of Information in Electronic Form (PCAOB 2023a) (the “proposal” or “release”).1 In the following sections, we provide our responses to the questions.
It is important to note that in our answers to Questions 1 to 12, we seek to offer suggestions that will improve the proposed standard and make it more complete, viewing the issuance of a new standard as a given. By contrast, when we subsequently consider the economic analysis underlying the proposed standard, we do not view new standard setting as a given. Rather, we consider how well the economic analysis establishes a solid foundation for new standard setting.
II. RESPONSES TO QUESTIONS
Question 1: Does the description of auditors’ use of technology-assisted analysis in designing and performing audit procedures accurately depict the current audit practice? If not, what clarifications should be made? Are there other aspects of auditors’ use of technology-assisted analysis that we should consider?
In general, we believe the proposal’s description of auditors’ use of technology-assisted analysis in designing and performing audit procedures accurately depicts the current audit practice, for the most part. For example, as described in Titera (2013) and the release, technology-assisted analysis can happen in any stage of an audit (from planning to reporting and from risk assessment to substantive tests). However, we believe there are multiple additional areas to consider from current practice that should be addressed in the proposed amendments.
First, the proposed amendments are designed to cover only the phases of designing and performing audit procedures. It is therefore missing guidance and examples regarding the use of technology-assisted analysis in risk assessments. Multiple research studies demonstrate that technology-assisted analysis improves audit quality in risk assessments (Wang and Cuthbertson 2015; Eilifsen, Kinserdal, Messier, and McKee 2020). Therefore, we encourage the PCAOB to amend the audit standards relating to technology-assisted analysis in risk assessments to meet the needs of current audit practice.
Second, technology-assisted analysis can allow auditors to conduct test of details on 100 percent of a population of transactions and to perform continuous auditing of balances (Issa, Sun, and Vasarhelyi 2016). Recent interview, survey, and case research also demonstrates that auditors use robotic process automation technologies to achieve enhanced efficiency when performing a test of details (Eulerich, Pawlowski, Waddoups, and Wood 2022). Therefore, the nature, timing, and extent of tests of details have drastically changed with technology-assisted analysis. In response to this change, we encourage the PCAOB to provide updated guidance regarding the nature and timing of tests of details.
Third, we believe that the proposed standard would be strengthened with guidance or examples regarding the procedures for data preparation and data validation. As the release notes on page 25, the standard is being amended “to address the risk that the external information maintained by the company and provided to the auditor to be used as audit evidence may be incomplete or inaccurate.” With the advancement of modern analytical tools, auditors obtain stronger capabilities and greater confidence in preparing and validating client data before conducting analyses (Moffitt, Rozario, and Vasarhelyi 2018). For example, auditors may use data preparation tools (e.g., Alteryx) to clean and join client datasets and then load them into audit analytics tools for testing (O’Brien and Stone 2021). In contrast, the current PCAOB standard does not provide guidance about how auditors should appropriately prepare and validate client data. Thus, we encourage the PCAOB to include guidance and/or examples regarding data preparation and data validation.
Fourth, although we agree that the release accurately describes most aspects of designing and performing audit procedures and that improvements within the standards are necessary, the scope of this amendment is limited. Specifically, page 5 of the release states, “The Board’s proposal is focused on addressing aspects of technology-assisted analysis and does not address other technology applications used in audits (e.g., blockchain or artificial intelligence) or the evaluation of the appropriateness of tools by the firm’s system of quality control” (emphasis added). We believe these italicized areas should be addressed within the standards as well. Both practitioners and academics have realized the significant impact of adopting artificial intelligence (AI) technologies in auditing. For example, in response to the recent popularity of generative AI tools (e.g., ChatGPT), PwC has announced an investment of $1 billion to expand and scale AI capabilities (PricewaterhouseCoopers (PwC) 2023). Both empirical results on AI investment and interview insights from audit partners show that the deployment of AI improves audit quality (A. Fedyk, Hodson, Khimich, and T. Fedyk 2022). However, research also finds negative consequences of AI adoption in auditing, such as algorithm aversion (i.e., discounting the advice from AI) (Commerford, Dennis, Joe, and Ulla 2022). Additionally, audit firm size can drive the degree of AI/robotics adoption, and robust adoption of technologies often happens only in larger audit firms (Bakarich and O’Brien 2021). Given the pros and cons of AI adoption in auditing, we believe it is necessary for the PCAOB to be forward-thinking and to regulate this area and enhance auditors’ confidence when leveraging the capabilities of AI in auditing.
Finally, we also encourage a minor clarification within the proposed amendments. In paragraph .10 A part (b), the release could clarify that the auditor would “test controls over the company’s procedures in part (a).”
Question 2: Does the release accurately describe aspects of designing and performing audit procedures involving technology-assisted analysis where improvements to PCAOB standards may be necessary?
Please refer to our response to Question 1.
Question 3: In addition to the proposed amendments, what other requirements may need to be included in PCAOB standards to address use of technology-assisted analysis in audits?
Please refer to our response to Question 1.
Question 4: Are the proposed amendments that clarify differences between tests of details and analytical procedures clear and appropriate? If not, what changes should be made to them?
The proposal indicates that the current standards only provide examples of substantive procedures and do not provide descriptions that differentiate between tests of details and analytical procedures. We agree that this lack of differentiation should be clarified by standard setters to ensure that sufficient appropriate audit evidence is collected, in general, and specifically within the context of this proposal (i.e., when using technology-assisted tools during tests of details).
As the release notes, auditors are using technology-assisted procedures for various procedures, including risk assessment, tests of details, and substantive analytical procedures. The amended standard seeks to clarify that tests of details include substantive procedures that examine individual items, whereas substantive analytical procedures typically do not (unless an individual item explains a significant difference within the procedure). We believe that this differentiation is appropriately clear in the amended standard. However, it is not clear how these particular changes to AS 1105 will necessarily “increase the likelihood that auditors obtain sufficient appropriate audit evidence when using technology-assisted analysis” (emphasis added). For example, the proposed amendments to .13 and .21 do not reference or differentiate between technology-assisted versus nontechnology-assisted procedures. These changes to the standards, therefore, do not resolve the question asked on page 14 about whether technology-assisted analysis can be a test of details and not an analytical procedure. We encourage the PCAOB to offer clarifying language or examples to paragraphs .13 to .21 to provide examples, context, and/or clarification for auditors when they use a technology-assisted analysis for either tests of details or analytical procedures.
Question 5: Would the proposed amendment that states that the relevance of audit evidence also depends on the level of disaggregation or detail of information necessary to achieve the objective of the audit procedure improve the auditor’s evaluation of the relevance of audit evidence? If not, what changes should be made?
The proposed amendment seeks to clarify AS 1105 to indicate that the relevance of audit evidence also depends on the level of disaggregation or detail of information necessary to achieve the objective of the audit procedure. Based on the changes being proposed, we believe the auditor’s evaluation of the relevance of audit evidence should improve for both technology-assisted and nontechnology-assisted procedures. The proposed amendment to .07 does not, however, mention or differentiate between technology-assisted versus nontechnology-assisted procedures. Based on the proposal’s intent to address the growing use of certain technology in audit procedures, we believe more clarification is needed on how this amendment specifically improves testing with technology-assisted procedures. We encourage the PCAOB to offer clarifying language or examples to paragraphs .07 and/or .13 to .21 to provide examples, context, and/or clarification for auditors when they use a technology-assisted analysis for either tests of details or analytical procedures.
Question 6: Are the proposed requirements that specify the auditor’s responsibilities when using audit evidence from an audit procedure to achieve more than one purpose clear and appropriate? If not, what changes should be made to the amendments?
The proposed standard seeks to update AS 1105.14 for auditors who use audit evidence from an audit procedure to achieve more than one objective. We believe that the proposed amendments are clear and appropriate. However, the proposed amendment to .14 does not mention or differentiate between technology-assisted versus nontechnology-assisted procedures. Based on the proposal’s intent to address the growing use of certain technology in audits, we believe more clarification is needed on how this amendment specifically improves testing with technology-assisted procedures. We encourage the PCAOB to offer clarifying language or examples to paragraphs .13 to .21 to provide examples, context, and/or clarification for auditors when they use a technology-assisted analysis for either tests of details or analytical procedures.
Question 7: Would the proposed amendments, which specify considerations for the auditor’s investigation of items that meet criteria established by the auditor when designing or performing substantive procedures, improve the identification and assessment of the risks of material misstatement and the design and implementation of appropriate responses to the assessed risks?
The considerations specified in paragraph .37A direct the auditor to consider the broader effects of investigating identified items. These effects may include raising questions about the original risk assessment, suggesting new risks of material misstatement, identifying misstatements or internal control deficiencies, or suggesting a need to modify the audit approach. We believe that highlighting these considerations is appropriate, as prior research indicates that auditors sometimes struggle to adequately respond to identified risks (e.g., Beasley, Carcello, Hermanson, and Neal 2013).
Regarding investigating identified items, the release states, “The proposed amendments would not prescribe the nature, timing, or extent of procedures for investigating the identified items” (22). Despite this language, there may be auditor uncertainty regarding the handling of large numbers of identified items. The proposed amendments should acknowledge that the auditor has to balance costs versus benefits in deciding what to test. For example, if thousands of items are deemed to be an exception in a test designated as a substantive procedure, the standard should indicate that the auditor should use professional judgment in deciding whether it is feasible to test all exceptions. Alternatively, the auditor should document the considerations in deciding what to examine further to obtain sufficient evidence.
Failing to explicitly acknowledge that the auditor has to weigh costs and benefits may encourage the auditor to forego analytics that may identify a large number of exceptions, particularly when using computerized techniques to test 100 percent of the transactions. Barr-Pulliam, Brown-Liburd, and Munoko (2022, 349) note that “uncertainty about regulators’ response and acceptance of emerging technologies can hinder its adoption.” Arguably, if auditors are uncertain as to how regulators will view the examination of exceptions identified as part of substantive testing, auditors may forego this potentially useful tool. Greater clarity needs to be included as to how the auditor will address large numbers of exceptions when using analytics as part of substantive testing or when testing 100 percent of items in a population.
Question 8: What other factors, if any, should the auditor consider when investigating items that meet criteria established by the auditor when designing or performing substantive procedures?
As noted in the response to Question 7, the standard should address costs versus benefits of testing 100 percent of exceptions. Further, the standard should acknowledge that auditors are not required to test 100 percent of exceptions if they can use alternative measures (sampling, isolating errors) to examine the exceptions.
Further, continuous monitoring/assurance could impact the nature, timing, and even scope of substantive testing. Barr-Pulliam et al. (2022, 348) indicate, “For example, instead of obtaining printouts of transactions from the client’s enterprise resource planning (ERP) for substantive testing, the emerging technology may require a direct connection to the client’s ERP for continuous monitoring/assurance. Client data security preferences and digitization capabilities influence auditors’ emerging technology deployment.”
Question 9: Are the proposed amendments that specify requirements for the auditor to perform procedures to evaluate the reliability of external information maintained by the company in electronic form that the auditor uses as audit evidence clear and appropriate? If not, what changes should be made to the amendments?
The standard does not directly address some issues that may be relevant. First is whether the auditor has to validate that the information created by others but maintained by the company is reliable when the company receives the information. A company may receive unreliable information but have strong controls over that bad information. Second, in addition to information maintained by the client, the auditor may use third-party information, for example, data from the Federal government, in analytics involving nonfinancial information. Does the auditor have a responsibility to validate the reliability of this type of information that was not created or maintained by the client? Arguably, the auditor would be relying on outside information without any comfort that the information is accurate. Finally, the appropriateness of the paragraph depends on whether the auditor should be responsible for controls over all information or only information related to financial reporting.
Further, in our response to Question 1, we discuss the need to provide guidance for data preparation and data validation (Moffitt et al. 2018; O’Brien and Stone 2021). These same issues of data preparation and data validation may apply to the process of evaluating external information maintained by the company in electronic form.
Question 10: Are the proposed amendments that emphasize the importance of controls over information technology for the reliability of audit evidence clear and appropriate? If not, what changes should be made?
Please refer to our response to Question 9.
Question 11: When the auditor uses information produced by the company and external information maintained by the company in electronic form, should PCAOB standards require internal controls over such information to be tested and determined to be effective for such information to be considered reliable audit evidence?
We believe that the Board needs to tread carefully in this area. Currently, the auditor has the responsibility for testing internal control over financial reporting (ICFR). Requiring the auditor to consider all information controls, depending on how far this extends, may significantly expand the scope of auditor testing. For example, information obtained from outsourced third-party vendors would seem to apply for this standard. Presumably, auditors now rely on System and Organization Controls (SOC) 1 reports by third parties that are relevant to ICFR. It is possible that this standard expands the reliance to include reviewing SOC 2 reports if those reports relate to nonfinancial information used by the auditor. Further, the Board may be inadvertently expanding the auditors’ scope to include controls over the collection of all information, not just that related to the financial statements. If there is a perceived need to test controls over all data outside of the system that creates financial data, we question whether such a change should come through audit standards.
Question 12: Are the proposed amendments that update certain terminology in AS 1105 clear and appropriate? If not, what changes should be made?
These definitions appear clear and appropriate. See also responses to Questions 4 to 6.
Economic Analysis: Other Research to Consider (Questions 13, 14, and 17)
In addition to the studies cited in the release, we call the Board’s attention to other recent research that may be useful as the Board continues to oversee auditors’ use of data analytics (DA). Specifically, we highlight selected studies in three areas: (1) the perceived impact of DA on auditing and financial reporting quality, (2) factors affecting auditors’ use of and reliance on DA, and (3) suggestions for optimizing auditors’ DA use. This literature relates to Questions 13, 14, and 17 in the release.
Perceived Impact of DA on Auditing and Financial Reporting Quality
Two studies provide evidence of the perceived positive effects of DA use on auditing and financial reporting quality. Kend and Nguyen (2020) conduct interviews and focus groups with auditing stakeholders in Australia. They find that stakeholders view the impact of DA on auditing as positive, in part because it provides auditors with more time to apply judgment in critical areas. The stakeholders also call on regulators to “keep on track with the fast-paced IT, automation evolution in the auditing field” (Kend and Nguyen 2020, 269). Saleh, Marei, Ayoush, and Abu Afifa (2023) conduct interviews of Canadian auditors and find evidence that auditors believe that DA use significantly improves financial reporting quality. In both studies, there is evidence that auditors and stakeholders perceive considerable benefits of DA use.
Factors Affecting Auditors’ Use of and Reliance on DA
Several studies examine issues related to auditors’ use of and reliance on DA. Jacky and Sulaiman (2022) analyze the content of comment letters submitted to the International Auditing and Assurance Standards Board’s Data Analytics Working Group. The authors find that many factors affect auditors’ use of DA, including “the usefulness of DA in auditing, authoritative guidance (auditing standards), data reliability and quality, auditors’ skills, [and] clients’ factors and costs” (Jacky and Sulaiman 2022, 31).
Cao, Duh, Tan, and Xu (2022, 131) examine auditors’ reluctance to rely on DA, in part due to a fear that inspectors “will second-guess the audit evidence gathered using DA” (see Gepp, Linnenluecke, O’Neill, and Smith 2018; Austin, Carpenter, Christ, and Nielson 2021). The authors conduct an experiment with Big 4 auditors, manipulating inspection risk as low or high and auditor mindset as “fixed” (auditors are focused on performance and being judged) or “growth” (auditors are focused on learning and improving). The authors find that “relative to low inspection risk, high inspection risk reduces auditors’ reliance on DA when auditors are prompted to adopt a fixed mindset but increases it when auditors are prompted to adopt a growth mindset” (Cao et al. 2022, 131). Thus, when inspection risk is high, the effect on auditor DA use depends on the auditor’s mindset.
Schmidt, Riley, and Swanson Church (2020) use a survey approach to understand accounting and finance professionals’ resistance to move beyond Excel and adopt DA. They find that the benefits of switching to DA and the perceived value of DA reduce DA resistance, whereas costs to switch to DA increase resistance.2
Koreff (2022) examines factors that affect auditors’ judgments when using DA. He conducts an experiment that manipulates whether the DA tool identifies anomalies or makes predictions and whether the data used by the DA tool are financial or nonfinancial. He finds that both the type of DA model and type of data affect auditors’ decisions regarding time budgets. Auditors increase time budgets more when financial data are used in predictive DA models and when nonfinancial data are used in anomaly DA models.
Barr-Pulliam, Brazel, McCallen, and Walker (2023) experimentally examine the effects of false positives and auditor rewards on auditor skepticism when using DA. Auditors are more likely to disregard DA results when false positives are high. Auditors are more likely to respond to DA-generated red flags when false positives are low, and auditors are consistently rewarded for being skeptical. Further, the authors find that when false-positive rates are very low, auditors tend to discuss the red flag with their manager before formally pursuing the red flag. Overall, the results suggest the importance of well-calibrated DA tools and consistent rewards for auditor skepticism.
Finally, Barr-Pulliam, Brown-Liburd, and Sanderson (2022) examine the effects of auditors’ DA through the lens of jurors’ assessments of auditor negligence. The authors conduct an experiment manipulating the opinion on ICFR (unqualified or adverse) and the audit testing method (statistical sampling or audit DA). The authors find that when the ICFR opinion is unqualified, jurors’ assessments of auditor negligence are lower when auditors use DA rather than statistical sampling.
Suggestions for Optimizing Auditors’ DA Use
Two recent papers offer insights for improving auditors’ DA use. No, Lee, Huang, and Li (2019) present the Multidimensional Audit Data Selection (MADS) framework to provide a systematic approach to DA use, including how to address a large number of outliers. The authors explain, “The MADS framework…[identifies] outliers based on multidimensional criteria and then prioritiz[es] the outliers to help auditors focus on the most problematic items while performing substantive tests of details” (No et al. 2019, 128).
Yoon and Pearce (2021) assess findings from 21 prior studies and offer their insights into auditors’ use of substantive analytical procedures, including procedures based on advanced analytics models. The authors note the limitations of certain substantive analytical procedures related to revenue, and they encourage complementary use of audit sampling and substantive analytical procedures.
Economic Analysis: Process (Questions 14 to 19)
The PCAOB has adopted a framework to conduct an economic analysis of all new and potential regulations. This framework has four main elements: (1) the need for the rule, (2) the baseline for measuring the rule impacts, (3) the alternatives considered, and (4) the economic impacts of the rule (and alternatives), including the benefits and costs (Public Company Accounting Oversight Board (PCAOB) 2023b). In submissions to prior proposed standards in 2023, we observed that the economic analysis had fallen short of this framework. This motivates our discussion of the economic analysis regarding technology-assisted analysis of information in electronic form.
One important clarification is in order with respect to this section of our response. In our answers to Questions 1 to 12, we seek to offer suggestions that will improve the proposed standard and make it more complete, viewing the issuance of a new standard as a given. By contrast, in this section, we consider the economic analysis underlying the proposed standard (including the need for standard setting to address technology-assisted analysis) at a higher level. In this economic analysis section, we do not view new standard setting as a given. Rather, we consider how well the economic analysis supports new standard setting.
The Need for the Rule and Alternatives Considered
The Board asserts that “advancements in technology have enabled auditors to expand the use of technology-assisted analysis in audits. If not designed and executed in accordance with PCAOB standards, audit procedures that involve analyzing information in electronic form with technology-based tools may not provide sufficient appropriate audit evidence” (PCAOB 2023a, 4). However, the proposed solution mainly clarifies existing requirements, highlighting that AS 1105, Audit Evidence (Public Company Accounting Oversight Board (PCAOB) 2010a), and AS 2301, The Auditor’s Responses to the Risk of Material Misstatement (Public Company Accounting Oversight Board (PCAOB) 2010b), apply to technology-assisted analysis of issuer information. These clarifications take the form of renaming “records and documents” as “information” and identifying that such information may be stored in an electronic format. The proposal also clarifies the differences between tests of details and analytical procedures. Although we expected a robust examination of alternatives, we agree that neither a separate technology proposal nor a data analysis definition is appropriate. Thus, one could argue that staff guidance would seem adequate to communicate the requirements of the two standards. However, as the Board has undertaken a modernization initiative to update older standards to reflect the current environment (Public Company Accounting Oversight Board (PCAOB) 2022a, 10), it appears reasonable to change the language around evidence (e.g., that “records and documents” become “information”).
Economic Analysis and Unintended Consequences
The Board asserts its research suggests the need to “more specifically address aspects of audit procedures that involve technology-assisted analysis” (PCAOB 2023a, 5). However, the release does not provide concrete, persuasive evidence of the need. Further, although economic analysis should consider both qualitative and quantitative impacts of proposed rulemaking, we do not observe either. The cited research is limited to a description of the tools used by large multinational audit firms (PCAOB 2023a, 28) or survey data describing how DA is or could be integrated into the audit (PCAOB 2023a, 29). The link between the proposed changes and the tool descriptions is missing for large multinational audit firms. The research on incorporating DA into the audit suggests that firms should be doing more. However, it is unclear why the Board would imply the need for greater DA adoption in its rationale yet remain silent in the standards themselves.
In prior proposals, the Board has made significant efforts to consider the scalability of its proposals for smaller firms. As the Board inspects over 200 audit firms and 800 engagements annually (Public Company Accounting Oversight Board (PCAOB) 2022b), it appears that a rich dataset exists to describe how the large firms are improperly using the approved tools. Alternatively, although the release describes survey data on the use of DA, the Board should have that data in the inspection files. The PCAOB’s economic analysis framework would suggest a rigorous analysis of these inspection files, detailing the nature and frequency of the misuse of technology tools or data analysis. Instead, the proposal speculates where and how auditors might become confused (e.g., “For example, currently some auditors might not appropriately investigate items identified when using technology-assisted analysis in designing and performing substantive procedures,” PCAOB 2023a, 41). Further, it is unclear why technology-assisted analysis is more prone to inappropriately designed multiple-purpose audit procedures than current practice.
We believe costs and benefits are idiosyncratic to each firm’s and engagement’s economics, and some may be skeptical that the current proposal will differ from the prior evidence standard on the likelihood of not obtaining sufficient appropriate audit evidence. The release’s focus on technology presents a potential unintended consequence. Although the proposed changes are not restricted to technology, auditors may view the modifications as increasing technology usage requirements. In addition, the proposal appears to suggest the Board is encouraging audit firms to adopt such technology, not to guide its use. The suggestion that firms might forgo using a technology that may negatively impact audit quality seems to suggest making tools required (PCAOB 2023a, 41). This apparent encouragement is troubling for two reasons. First, as the Board has not quantified costs or benefits, there is no basis for evaluation. Second, the supposition that efficiencies would accrue to the firms, potentially impacting audit efficiencies or even audit fees, is beyond the Board’s charge of improving audit quality. Instead, we would expect the Board to be agnostic about auditors’ decisions regarding the tradeoffs of technology usage, instead focusing on the objectives of the audit. Whether an auditor uses technology is not a market failure requiring a regulatory solution.
Overall, we continue to have concerns about the economic analyses of proposed audit standards. In this case, it seems that the clarifications made could have been achieved more efficiently. Having said that, our answers to Questions 1 to 12 are designed to improve the proposed standard and make it more complete, given that standard setting is the approach that the PCAOB is taking in this area.
REFERENCES
We adapt or use language from PCAOB (2023a) and other PCAOB resources.
Also see Dagiliene and Kloviene (2019) for evidence from the Lithuanian context on factors affecting auditors’ use of DA.