SUMMARY
Bierstaker, Hanes-Downey, Rose, and Thibodeau (2018) investigate whether using a fraud story (a one-page summary of the key facts in an SEC AAER written as a narrative) as compared to a traditional checklist improves fraud risk assessments in two separate experiments. This article summarizes their findings and discusses practical implications and actionable suggestions for audit practitioners. Specifically, the summary focuses on the capacity of fraud stories to help novice auditors develop knowledge structures that closely resemble the knowledge structures of experts and ultimately to improve experienced auditors' risk assessments. Importantly, one Big 4 firm that participated in this research has adopted new and innovative fraud training methods based upon the results of this study. This article discusses these training implications, along with the potential for stories to help auditors improve their fraud-related judgments and additional considerations for the design of decision aids and knowledge management systems.
I. INTRODUCTION
Professional standards require auditors to assess and respond to the risk of material misstatement due to fraud on each audit. Despite this requirement, auditors rarely experience the discovery of an actual financial statement fraud in practice (ACFE 2014). The lack of direct experience with actual fraud cases results in auditors having limited fraud-related knowledge to draw upon when assessing and responding to fraud risks, which can potentially increase regulatory and engagement risk for audit firms. Thus, it is important to understand how to help auditors develop fraud-related knowledge and improve their risk assessments, particularly for junior auditors who perform most of the fieldwork.
To supplement auditor knowledge and improve judgment performance, audit firms routinely employ decision aids, such as task-specific checklists. However, prior research cautions that decision aids (like checklists) and some structured audit technologies can inhibit critical thinking and professional skepticism (Boland, Daugherty, and Dickins 2019; Rose and Wolfe 2000). In “Effects of Stories and Checklist Decision Aids on Knowledge Structure Development and Auditor Judgment” (Bierstaker, Hanes-Downey, Rose, and Thibodeau 2018), the authors propose that a knowledge tool, consisting of a story about an actual fraud, will help novice auditors to develop valuable fraud knowledge structures. In addition, the authors suggest that stories can improve the judgment processes used to make fraud risk assessments and staffing judgments. The purpose of this summary is to review the findings of Bierstaker et al. (2018) and discuss the implications for the design of training materials, decision aids, and knowledge management systems to be used in practice.
II. EXPERIMENT ONE
To measure how novice auditors organize information about fraud in memory, Bierstaker et al. (2018) asked senior auditing students to complete a “fraud cues” rating task that was designed to capture each participant's existing knowledge structures related to fraud. The knowledge structures reveal how auditors organize fraud knowledge in long-term memory. Following the measurement of each participant's knowledge, the auditing students received either: (1) a fraud story; (2) a checklist based on the risk factors contained within the fraud story; or (3) a checklist based on the risk factors contained within the professional auditing standards. The checklists represented the current practice of using a standardized and checklist-driven approach to fraud risk assessment. The fraud story was based on a past Accounting and Auditing Enforcement Release (AAER) and was vetted with several public accounting partners and senior managers. After studying the fraud story or checklist, participants completed the same rating task again to measure whether the story or checklists caused any changes to the way they organized knowledge about fraud.
Based upon individual participants' responses to the rating task, Pathfinder network scaling (see J. Rose, McKay, Norman, and A. Rose 2012) was used to create a graphical display of each participant's organization of their knowledge about fraud before and after using either the fraud story or one of the checklists.1 Pathfinder network scaling was also used to develop a knowledge structure of experts (based on the knowledge of a Big 4 partner and accounting professor, both with extensive fraud experience). The Pathfinder technique made it possible to compare the organization of fraud knowledge in memory for both novices and experts both visually and statistically. A total of 102 novice auditors (i.e., senior accounting students about to enter practice) participated in the experiment.2
III. RESULTS—EXPERIMENT ONE
Figure 1, Panel A illustrates how reading a fraud story caused novice auditors to reorganize their knowledge related to fraud in such a way that their knowledge structures became more similar to an expert's knowledge structure than did using either of the two checklists. In essence, this finding reveals that the novice auditors who read a fraud story started to organize their knowledge more like an expert auditor, as compared to the auditors using either of the two checklists. Participants using the checklists did not experience similar improvements in the organization of their knowledge about fraud. In fact, the participants who used the checklist based on risk factors in the professional standards reorganized their knowledge of fraud such that their knowledge structures were even less like an expert knowledge structure after using that checklist. Results illustrated in Figure 1, Panel B additionally show that reading a fraud story caused the greatest change in the knowledge structures of novice auditors. These findings are important for two reasons. First, they identify a potential problem created by using common checklists when completing audit tasks. Second, they demonstrate the mechanism by which a fraud story can lead to enhanced auditor performance when assessing fraud risks. The judgment performance of experienced auditors is the focus of the second experiment.
FIGURE 1 Knowledge Structure Development
IV. EXPERIMENT TWO
Experiment two examined the potential for fraud stories to improve experienced auditors' professional judgments relative to the use of traditional checklists. Although checklists are commonly used by audit firms to improve auditor judgment, audit quality, and decision making, prior literature finds mixed results regarding their effectiveness related to fraud (e.g., Pincus 1989; Asare and Wright 2004; Wilks and Zimbelman 2004; Rose et al. 2012). In contrast to checklists, stories provide rich context, and this context has the capacity to trigger memory searches that promote deeper understanding of risks. That is, stories have the capacity to activate effective judgment strategies.
In experiment two, experienced auditors were assigned to finalize the planning phase of a hypothetical audit of a medical company, which included assessing overall engagement risk and fraud risk at the account level. The case provided participants with background and summary information on the company and also described the audit and audit history, the control environment, and relevant account information. Interwoven into the details of the case were risk factors that the authors vetted with several public accounting partners and senior managers. In addition to the case, auditors received either a fraud story, no additional information (i.e., a control condition), a checklist based on the risk factors contained within the fraud story, or a checklist based on the risk factors contained within the professional auditing standards.
Auditors' assessments of overall engagement risk and fraud risk were evaluated against assessments made by an expert panel of audit partners. Specifically, three Big 4 partners experienced with discovery of financial statement fraud and with audit experience ranging from 14 to 35 years worked through the case without any decision support (story or checklist) and completed the same risk assessments as the auditors in the experiment.3
V. RESULTS—SECOND EXPERIMENT
One hundred fifty-four senior audit associates from two Big 4 firms participated in the second experiment. The auditors had a mean audit experience of 45.1 months and a mean experience with material statement fraud of 1.69 (on a scale of 1 to 7). Thus, consistent with the findings of ACFE (2014), most participants lacked fraud experience.
Fraud Risk
The risk of financial statement fraud (FRAUD_RISK) was rated on a scale of 1 (low) to 7 (high). The average FRAUD_RISK assessed by the expert panel was 5.67, and this was used as the benchmark to represent a high-quality judgment for the case. As shown in Figure 2, the average FRAUD_RISK rating for participating auditors who read the fraud story was 4.75, compared to 3.92 for those who received no decision aid, 3.70 for those with a checklist based on the fraud story, and 4.15 for those with a checklist based on professional standards. Auditors in the story condition rated FRAUD_RISK significantly higher than the other groups (i.e., the no decision aid group and the groups using either checklist); a higher risk assessment was appropriate for this case. Further, multivariate statistical tests that control for participants' experience with financial statement fraud indicate that auditors who received a fraud story rated FRAUD_RISK significantly closer to experts' ratings than did auditors using either checklist. Planned contrasts, statistical comparisons of the difference in the average risk assessment by each group and the expert panel, illustrate that the differences between the risk assessments of auditors reading a fraud story and the expert panel were significantly smaller than any of the other groups.
Overall Engagement Risk
Overall engagement risk (OVERALL_RISK) was rated on a scale from 1 (low) to 7 (high). The average OVERALL_RISK assessed by the expert panel of audit partners was 5.67. The average OVERALL_RISK rating for participants with the fraud story was 4.33, compared to 3.49 for those receiving no decision aid, 3.65 for those with a checklist based on the fraud story, and 3.75 for those with a checklist based on professional standards (see Figure 2). Comparisons of these means show that auditors reading a fraud story rated OVERALL_RISK significantly higher than each of the other groups. In addition, participants receiving a fraud story rated OVERALL_RISK closest to the expert panel, both in planned comparisons of the average difference in overall risk assessed from the expert panel as well as multivariate tests controlling for fraud experience.
VI. CONCLUSIONS AND IMPLICATIONS
Bierstaker et al. (2018) find that reading a fraud story helps novice auditors to develop knowledge structures that are similar to experts' knowledge structures. A fraud story also improves the fraud risk and engagement risk assessments of experienced auditors. Given that auditors rarely experience financial statement fraud, and checklists appear to be relatively ineffective for promoting knowledge structure development or improving risk assessment, this study may help firms to design better decision aids, knowledge management systems, and training materials.
Several specific considerations are relevant to the value of fraud stories (i.e., a narrative context) as a decision aid for use in audit practice. First, audit firms are required to develop and employ staff that are competent and capable of fulfilling their responsibilities in the field, including assessing and addressing the risk of fraud (AICPA 2002, 2012). Given that auditors typically lack experience with financial statement fraud, and regulators continue to highlight concerns in this area (PCAOB 2007, 2008, 2014), firms are working to help improve their auditors' fraud-related judgments. Several deployment opportunities for stories exist that may support firms in these efforts and ultimately improve auditors' mental models of fraud. For example, audit firms could consider the possibility of creating a library of relevant fraud stories within the firm's electronic workpaper platform to be accessed by audit professionals when needed. If a member of the engagement team has a particular concern about a fraud risk, they could read an industry specific story about a fraud. By embedding industry-specific fraud stories in their audit applications, auditors could read short stories in the field on demand (e.g., before a brainstorming session or prior to audit planning activities) to facilitate the development of more expert-like mindsets about fraud during key phases of the audit. Future research could further guide such implementation by determining when and how often auditors should read such stories to maintain the benefits for their knowledge structures and risk assessments.
The use of fraud stories in this regard is particularly important when considering that checklists continue to be routinely used by auditors to assess fraud risks, even though prior research questions the effectiveness of checklists as a decision aid (e.g., Rose and Wolfe 2000). The results of the Bierstaker et al. (2018) study make clear that a fraud story can yield knowledge structures and risk assessments that are closer to an expert level, whereas checklists can have the opposite effect. Thus, although audit firms often continue to use checklists, this research demonstrates their potential to make auditors' knowledge structures less similar to experts' knowledge structures, which is concerning given that use of checklists and other forms of standardization are increasingly favored by firms when responding to PCAOB deficiencies (Boland et al. 2019). We suggest that an increase in the use of fraud stories, or stories more generally in complex areas where auditors lack experience, could help to mitigate some potential threats of using checklists.
Interestingly, one participating firm has already employed the results of the Bierstaker et al. (2018) research to integrate fraud stories into their national training methodology for audit staff, which may be helpful for other firms to consider. Specifically, the use of stories may provide a rich set of training contexts for auditors to help organize newly acquired knowledge to levels comparable to an expert. For some time, education and developmental psychologists have argued that reading a short story about an event with which the reader has little or no prior experience will cause active mental questioning and lead to the development of knowledge that helps the brain to explain the event. Such memory patterns allow new information to be understood and effectively applied to subsequent judgments (see Schank 1986) and may be particularly useful to audit firms as a formal training tool.
There may be other areas where fraud stories can also be deployed in a productive manner. For example, beyond purely domestic audits, the use of stories as a training tool may be helpful in providing context to component auditors who can be unfamiliar with how risk factors present themselves in U.S. issuer audits, or who lack a common understanding with the group auditor. Indeed, lengthy instructions, templates, and checklists are often used extensively in group audits (Hanes Downey and Westermann 2021). Although further research is needed, stories would appear to have the capacity to improve the knowledge structures of component auditors who assess risks and perform work locally on behalf of group auditors in the U.S., which is another key area of concern for regulators (PCAOB 2017, 2019). Finally, it is unknown to what extent stories and checklists could complement each other. Future research could investigate if the comprehensive nature of checklists could be augmented by the experiential based approach of the fraud story.
REFERENCES
Pathfinder network scaling is a method used by researchers to measure the interrelations of ideas and concepts in a person's long-term memory, which reveals how the knowledge is organized in memory (see Bierstaker et al. 2018).
See the full academic study for details about the administration of the experiment and the methods used to measure participant knowledge structures (Bierstaker et al. 2018). Students were employed in experiment one given their comparability to staff auditors (who lack experience with financial statement fraud) and the infeasibility of completing the knowledge structure rating task within the time allotted by participating firms for experiment two.
See Bierstaker et al. (2018) for more details about the administration of the experiment.