Comments of the Auditing Standards Committee of the Auditing Section of the American Accounting Association on the PCAOB’s Proposed Auditing Standard—Designing and Performing Substantive Analytical Procedures and Amendments to Other PCAOB Standards Free

The participating members of the Auditing Standards Committee of the Auditing Section of the American Accounting Association are pleased to provide comments on the Public Company Accounting Oversight Board’s (PCAOB) Proposed Auditing Standard—Designing and Performing Substantive Analytical Procedures and Amendments to Other PCAOB Standards (PCAOB 2024a). We applaud the PCAOB (the Board) for its efforts to improve audit quality by revising existing standards for performing substantive analytical procedures (SAPs), as well as amendments to other PCAOB standards. This letter presents the participating committee members’ comments on the proposal (PCAOB 2024a).¹

This comment letter responds to selected questions from the proposal and is structured as follows: Section II presents our committee’s overall observations on the PCAOB’s proposal, Section III provides key takeaways from academic research, Section IV considers perspectives on the PCAOB’s economic analysis, and Section V provides our conclusion.

We commend the Board for its efforts to revisit the standard on SAPs. We support the goal of updating the standard in response to technological changes, as well as inspection deficiencies that suggest the current standard may require clarification. While we support these objectives, we have identified some concerns with the proposal. Our overall observations about the proposal relate broadly to (1) potential adverse effects on audit quality and (2) the need for additional guidance, in part to manage audit firms’ inspection risk.

With respect to the potential effects of the proposal on audit quality, we have three primary concerns. First, we believe that the proposal will further reduce auditors’ use of SAPs, to the potential detriment of audit quality. The proposed standard will certainly make it easier for the PCAOB to inspect auditors’ use of SAPs. Previous studies (Trompeter and Wright 2010; Glover, Prawitt, and Drake 2015) indicate that SAPs present auditors with substantial inspection risk, which may increase under the proposed standard. With respect to audit quality, Glover et al. (2015) indicate that dropping SAPs in favor of additional detailed testing serves to shift work from more senior auditors to more junior auditors, and detailed testing may focus only on routine transactions and not riskier “top side” entries. Thus, it is not clear that a shift away from SAPs toward detail testing will improve audit quality; it may, in fact, decrease audit quality. We believe the Board should be explicit about its intentions with this standard. Is the goal to enact a de facto “ban” on SAPs? If so, is there reason to believe that such a ban will enhance audit quality?

Second, the proposal would narrow the definition of SAPs so that such procedures could only be used to obtain a high level of assurance. Specifically, the proposal requires auditors to set the threshold for investigation of differences at or below tolerable misstatement. Under the proposal, auditors will employ SAPs only when they are sufficient to provide persuasive evidence regarding a relevant assertion. The apparent elimination of the ability to use less precise SAPs to obtain complementary substantive evidence (i.e., in situations where analytical procedures are used in combination with substantive tests of details) could serve to reduce audit quality. Less-precise SAPs can offer important evidence to the auditor when used in combination with other testing. Specifically, Glover et al. (2015) present a compelling argument for using SAPs as a complementary test that can help to identify material misstatements, even when the threshold for investigation is large. By eliminating the ability to achieve lower levels of assurance through SAPs (i.e., using thresholds greater than tolerable misstatement), the standard may adversely impact audit quality, as fraudulent misstatements are often many multiples of materiality.

We believe the Board should reconsider the elimination of the concept of levels of assurance and provide guidance as to how SAPs may be combined with other testing to achieve a high level of assurance. Glover et al. (2015) provide an excellent roadmap for this purpose. The proposal seems to assume that SAPs are done in the absence of other testing; however, this is typically not the case (Glover et al. 2015).

Finally, the proposal outlines a process for investigating and resolving differences between the auditor’s expectation and the company’s amount. The proposal seems to require the auditor to resolve unexpected differences by revising the auditor’s expectation until any difference from the company’s amount is below the threshold for investigation, clearly determining that there is a misstatement, or dropping the SAP entirely and replacing it with other testing. In other words, the SAP ultimately “works” and provides evidence supporting the company’s amount, provides clear evidence of a misstatement, or does not work and is eliminated (possibly not even appearing in the workpapers–see discussion below). This raises two issues.

First, there is little focus on exactly how the auditor should evaluate unexpected differences. Anderson and Koonce (1998) describe a process in an analytical review setting where the auditor examines unexpected fluctuations by investigating their cause(s). Anderson and Koonce (1998) emphasize the importance of plausibility checking and sufficiency checking. Thus, the auditor considers whether the company’s explanation is plausible (i.e., Does it make sense, and does other information support it?), and then also considers whether the explanation is sufficient to account for the magnitude of the unexpected fluctuation. Paragraphs 9–12 of the proposed standard could be enhanced if they incorporated the Anderson and Koonce (1998) guidance for evaluating the plausibility and sufficiency of explanations. The Anderson and Koonce (1998) approach better reflects what the auditor learned through performance of the analytical procedure and enables the auditor to better document the effort involved in performance of the procedure.

Second, we wonder if the approach in the proposal could promote confirmation bias by the auditor trying to get the procedure to “work,” which would decrease audit quality. The auditor seemingly “needs” to get close to the company amount, determine clearly that there is a misstatement (presumably with additional testing), or discard the analysis. Obviously, getting the procedure to work offers time budget and client relations advantages, thus potentially leading the auditor to confirm the company amount. We believe that care needs to be taken not to unintentionally promote confirmation bias.²

Although we appreciate the proposal’s principles-based approach, we believe there is a need for additional guidance and examples. A principles-based framework allows for auditor flexibility and adaptability, but in a world of very stringent PCAOB inspections and enforcement, auditors may not be willing to use SAPs without specific guidance and examples (e.g., Trompeter and Wright 2010; Glover et al. 2015).

First, the proposal could use examples of appropriate uses of SAPs, including documentation that would be acceptable to PCAOB inspectors. In our view, two particular areas require more specificity: (1) determining plausible and predictable relationships, and (2) documenting SAPs that do not “work.” Regarding plausible and predicable relationships, what type of analyses and documentation does the Board envision? Analyses to support such relationships could range from simple observations based on a partner’s industry experience to many quarters of regression models to advanced machine learning applied to Big Data. Evidence of predictability also could range widely, especially in an unstable and often unpredictable world—over what period must something be “predictable”? Overall, auditors are likely to be concerned about hindsight bias in inspections, where inspectors’ hindsight may make it obvious that a relationship was no longer to be expected.³ Likewise, there is a need for clarity on documenting the examination of differences between auditor expectations and company amounts, including whether/how to document SAPs that do not “work.”

Second, we believe that the proposal could offer more guidance and rich examples related to auditors’ use of nonfinancial measures. Brazel, Jones, and Zimbelman (2009); Brazel, Jones, and Prawitt (2014); and Brazel and Schmidt (2019), among others, offer insights in this area. Also, see evidence of the successful use of nonfinancial measures, such as consumer tweets (Rozario, Vasarhelyi, and Wang 2023) and weather (Yoon, Kogan, Vasarhelyi, and Pearce 2024) in analytical procedures.

Finally, the proposal is silent on auditors developing a range for an expectation (e.g., $1.38 million to $1.48 million, with X percent confidence), suggesting that auditors will develop single point estimates (e.g., exactly $1.43 million). We believe the proposed standard should be explicit about the appropriateness of ranges, including how such ranges should be used. For example, should auditors look to AS 2501.25 and AS 2810.13 for resolving unexpected differences?

As noted by the Board, the current AS 2305 was originally drafted by the AICPA (i.e., SAS 56) and has been largely unchanged since 1989. During the past 35 years, several academic research studies have explored various aspects of auditors’ use of analytical procedures, principally focusing on the risk assessment and substantive testing stages of the audit.⁴ Although the core principles underlying the design and performance of SAPs remain essentially unchanged today, the Board rightfully acknowledges the need to address recent technology developments, enhanced data availability, and potential risks and impediments to the proper use of SAPs.

Although the Board has identified and considered the findings from a number of relevant academic studies focused on the design and use of SAPs, we believe there are several other academic studies related to core SAP principles and related matters that the Board should consider.

As the Board notes in its economic analysis and discussed in this letter, the use of SAPs declined throughout the 2010s. Consistent with discussions throughout the Board’s economic analysis, recent anecdotal evidence suggests that the use of SAPs has yet to rebound, particularly among nonaffiliated firms.⁵Glover et al. (2015) suggest this could be a persistent effect of prior efforts by the PCAOB that discouraged auditors from performing appropriately rigorous SAPs. As emerging technologies allow auditors to perform increasingly sophisticated analytics, any hesitation among auditors to use appropriately rigorous SAPs will lead to increasingly suboptimal outcomes for investors. We are therefore concerned that the potential benefits of a new standard will be limited unless the standard leads audit firms (and audit teams) to increasingly view SAPs as reliable and acceptable sources of evidence.

The Board notes that the Proposed Auditing Standard is principles-based. We observe that the proposal has no illustrative examples, no implementation guidance, and makes no mention of technology, disaggregated data, visualization, patterns, anomalies, or other terminology relevant and specific to modern applications of SAPs. We believe the stark nature of the proposal will create ambiguity (and/or increase existing ambiguity) around what practices the Board will consider acceptable versus unacceptable during the inspection process. This ambiguity could result in lower utilization of appropriately rigorous SAPs—and, ultimately, harm investors.

With these concerns in mind, we encourage the Board to consider academic studies that provide insight into ways to help auditors re-embrace appropriately rigorous SAPs. In a recent study, Austin, Carpenter, Christ, and Nielson (2024) find that empowering auditors helps them to overcome constraints and produce higher-quality work. Illustrative examples and implementation guidance can empower auditors to overcome ambiguity inherent in a principles-based standard around a complex and highly subjective audit task.

We also encourage the Board to consider academic studies that examine how imprecise accounting standards affect auditors. Principles-based auditing standards are similar to imprecise accounting standards in that both increase auditor exposure to criticism and liability. Namely, research shows that imprecise accounting standards can increase auditor liability (e.g., Gimbar, Hansen, and Ozlanski 2016) or increase firms’ risk management costs (e.g., Grenier, Pomeroy, and Stern 2015; Backof, Bamber, and Carpenter 2016). However, unlike imprecise accounting standards (which auditors are required to apply when their clients enter related contracts), auditors can effectively “opt-out” of any perceived ambiguity in the proposal by electing not to employ SAPs and increasing their reliance on tests of details. Again, robust illustrative examples and implementation guidance (as well as clear expectations around the use of technology) could help increase the precision of the Proposed Auditing Standard—and thereby facilitate auditors’ use of appropriately robust SAPs, in turn improving audit quality and protecting investors.

The Board acknowledges the modern benefits of enhanced data availability, including the auditors’ ability to leverage technology tools to examine up to 100 percent of a target population, yet the proposal appears agnostic to technology. Academic research provides some findings that may inform the standard and/or regulators’ guidance on the use of technology tools.

As technology tools and data availability increasingly present the potential to move from traditional sampling for substantive tests to performing SAPs that integrate data from an entire population, standards on SAPs should acknowledge and address this potential. In a recent study, Ballou, Grenier, and Reffett (2021) describe a series of experiments wherein they find that common external stakeholders (i.e., individual investors, jurors, peer quality-control reviewers) are generally receptive to and comfortable with SAPs based on population testing, suggesting that perceived audit quality may be enhanced by the design and performance of SAPs that incorporate entire populations in the expectation model. Relatedly, Barr-Pulliam, Brown-Liburd, and Sanderson (2022) find that jurors perceive population testing as a sign of higher audit quality, making them less likely to find auditors negligent when there is an audit failure. However, jurors do not perceive a difference in assurance levels between advanced data analytics that enable population testing and traditional sampling methods.

A. Rose, J. Rose, Sanderson, and Thibodeau (2017) find that data visualizations based on large data sets are more effective when viewed after reviewing more traditional substantive audit evidence. In addition, a working paper by Anderson, Hobson, and Peecher (2020) suggests that the categorization of rich data visualizations (i.e., risk assessment versus SAPs) can influence how auditors interpret and respond to the data. Another study by A. Rose, J. Rose, Rotaru, Sanderson, and Thibodeau (2022, 53) finds that data visualizations produce different cognitive and emotional responses in auditors, possibly enhancing “auditors’ ability to recognize disconfirming evidence.” Thus, regulators may wish to consider the potential benefits of encouraging auditors to employ best practices based on research findings to integrate the use of data visualizations in the design and performance of SAPs.

In their studies, Appelbaum, Kogan, and Vasarhelyi (2017) and Yoon and Pearce (2021) comment on the potential for exogenous and unstructured data (e.g., text interpretation, voice recognition, video/picture recognition) to provide additional insights and value to auditors’ SAPs. Yet, the proposal fails to provide examples or guidance to auditors seeking to incorporate such data into their expectation models for SAPs. Regulators may need to empower auditors to incorporate these types of data by referencing them and prescribing or acknowledging best practices and potential strengths and risks of incorporating this data into SAPs.

The proposal specifies that SAPs “are appropriate only when designed and performed…at a level of precision sufficient to respond to an assessed risk of material misstatement” (.01), yet the standard does not explicitly tie the assessed risk of material misstatement to the design and performance of the SAP. Consistent with research by Backof, Bowlin, and Goodson (2022), regulators may wish to prescribe specific ways that auditors can tie their use of SAPs–either alone or in combination with other audit procedures—to assessed levels of risk of material misstatement. Given the findings by Rose et al. (2017), regulators may emphasize the benefits of examining evidence from SAPs after results from other substantive tests when addressing higher-risk assertions or accounts where SAPs alone may not be sufficient.

We applaud the Board for explicitly stating that auditors may not develop their expectations “using the company’s amount or information that is based on the company’s amount” (.07). Given common psychological biases (e.g., anchoring bias, confirmation bias), it is important that expectations developed in SAPs are more than a replication of the client’s analysis. In documenting associated risks, academic research has found that auditors tend to test details of management’s estimate rather than using approaches that rely less on management-provided information (e.g., developing an independent expectation) (e.g., Griffith, Hammersley, and Kadous 2015) and anchoring on management’s numbers. In these situations, Earley, Hoffman, and Joe (2008) describe management as the “first mover” and the auditor as the “second mover.” This posture as the second mover makes auditors susceptible to the “curse of knowledge bias,” particularly when information from management is incorrect or otherwise biased.

Since November 2022, the Board has proposed nine and adopted five auditing standards. Our committee’s comment letters on these previous proposals have consistently presented detailed concerns about the PCAOB’s limited economic analysis, which we viewed as often inconsistent with the PCAOB’s established economic analysis policies. Therefore, in this instance, we commend the Board’s efforts for strengthening its compliance with its economic analysis framework, which includes: “(1) the need for the rule, (2) the baseline for measuring the rule impacts, (3) the alternatives considered, and (4) the economic impacts of the rule (and alternatives), including the benefits and costs” (PCAOB 2024b).

In the current proposal’s economic analysis, the PCAOB identifies the following areas to examine with their economic analysis framework: “(i) the PCAOB staff’s analysis of audit firm methodologies; (ii) firms’ use of technology-based tools when designing and performing SAPs; (iii) additional observations from PCAOB oversight activities; (iv) academic literature discussing SAPs; and (v) activities of other standard setters” (PCAOB 2024a, 35–36).

It is encouraging and appropriate that the proposal’s economic analysis examines audit methodologies to evaluate compliance with current auditing standards, PCAOB oversight activities to ensure an understanding of the scenarios in which auditors are lacking when testing SAPs, and activities of other standard setters to ensure greater consistency. Section III also provides our discussion of the academic literature for SAPs. Overall, we appreciate the Board’s efforts to identify the “distance” firms may have to go to comply with the new standard.

We ask that the Board continue to consider the impact of all newly-proposed standards on small to medium firms. Consistent with previous comment letters, we have similar concerns regarding the increased cost that this new regulation can have on small to medium firms. In that vein, regarding the use of technology, the proposal acknowledges that “the use of more sophisticated data analysis tools has become more prevalent, at least among larger firms, including tools used to design and perform substantive analytical procedures” (PCAOB 2024a, 37) and “technology-based tools can enable auditors to disaggregate data to a level where the most plausible and predictable relationships are more readily identified, which in turn can improve the precision of the substantive analytical procedure (e.g., by improving the precision of the auditor’s expectation)” (PCAOB 2024a, 38). However, small to medium firms may not possess the resources to develop or utilize such technological tools to efficiently obtain the required relevant and reliable audit evidence. Further, we find it disconcerting that the PCAOB suggests that when “the perceived cost of performing compliant SAPs is too high, auditors would choose to use tests of details instead” (PCAOB 2024a, 50). Therefore, we continue to encourage the PCAOB to consider the impact of their new standards on small to medium firms to ensure a robust and competitive marketplace for audit firms.

Notwithstanding positive strides in compliance with the PCAOB’s economic framework, we encourage the Board to revisit its economic analysis to develop a quantitative assessment. Without such an analysis, it is impossible to determine whether the benefits (intended and unintended) exceed the costs specific to this proposal and cumulatively with the numerous previously proposed and adopted standards. Notably, five auditing standards will become effective within twelve months. Given the volume and speed of the Board’s recent standard setting, each new proposal should consider the economic costs of each proposal and the cumulative costs. The Board may find that the marginal costs of this proposal may be absorbed in the cumulative costs of implementing the recently adopted new standards. Alternatively, the costs of this latest proposal may distract from the Board’s goal of improving audit quality. We continue to encourage the Board to live up to its responsibilities to demonstrate why the marginal benefits of improved audit quality justify the marginal costs of implementing and maintaining changes in the auditing standards. Taken together, we commend the PCAOB for its efforts to strengthen its evaluation of the economic impacts of its proposals.

We applaud the PCAOB for its ongoing commitment to improve audit quality by addressing enhancements to SAPs. Based on our committee’s assessment of the proposal, we provided some overall observations, an analysis based on academic research, and perspectives on the proposal’s economic analysis.