Super Shears: Micro Cases to Reduce Confirmation Bias in Evaluating Management Estimates Free

Auditing accounting estimates has been cited by the PCAOB as one of the three broad areas of recurring audit deficiencies (PCAOB 2015): (1) auditing internal controls, (2) assessing risk of material misstatement, and (3) auditing accounting estimates. The two micro cases seek to improve skill in the last of these three areas of concern: evaluating evidence for management estimates. The PCAOB's 2015 Staff Bulletin summarizes the concern:

These two micro cases provide an opportunity for students and audit professionals to practice completing a difficult auditing task—auditing management estimates. The case facts indicate that the client's product strategy has changed and therefore the prior year's practices may be insufficient for this year's accrual. Of the evidence gathered, some does support and some does not support management's estimates. Students must identify the supporting and disconfirming evidence and then decide if, on balance, the financial statement amounts are fairly stated.

While the PCAOB's broad areas of deficiencies relate primarily to complex estimates (PCAOB 2015), the data provided in these cases indicate that auditors, and not just students, could benefit from practice in evaluating arguments for and against management's representations for even less complicated estimates. These materials are particularly unique in that they both assess and then improve the skill levels, providing the motivation from the initial case to concentrate on and discern how to tease out arguments that support or cast doubt on audit assertions.

The brief and compact nature of these micro cases provides flexibility for instructors because they are appropriate for in-class and/or homework assignments, and for both face-to-face and online classes. The cases can be used for practice and/or assessment by auditing students and practicing auditors.

These realistic tasks give participants an appreciation for the difficulty of determining and auditing client estimates, without being overly difficult for novices. Furthermore, by asking students and auditors to evaluate contradictory evidence, the cases offer practice on important skills that are part of evaluating evidence and audit judgment—abilities called for by the profession for decades and specifically needed by auditing professionals (Nelson, Ratliff, Steinhoff, and Mitchell 2003; PCAOB 2010; Griffith, Hammersley, Kadous, and Young 2015; PCAOB 2015). The auditor participants report that the cases promote skeptical thinking by motivating a questioning mindset about estimates, and the evidence indicates that auditors improve their ability to detect disconfirming evidence after practicing on the first case.

Your professional career has commenced! You are the staff auditor for a CPA firm that audits Super Shears, a firm that manufactures and sells the market-leader electric clippers used to groom dogs. Until the past year, Super Shears' clippers were sold exclusively to commercial grooming shops, and offered a three-year warranty on all products sold. In general, the pet industry has been strong, even during the recession. But Super Shears' commercial customers were hurt when dog owners who were impacted by the recession learned to groom their own dogs at home. In order to improve sales, Super Shears introduced a “home version” of their clipper. This product is cheaper because it needs fewer metal parts and less substantial blades, casing, and cording. Home groomers typically use the clippers six to ten times each year, about as much as professional groomers use the clippers in a single day.

You have been assigned to audit a routine management estimate: the allowance for uncollectible accounts. The client has provided the lead sheet, and a member of your audit staff, who was pulled to work on another job, has completed the audit program and gathered considerable evidence regarding the estimate. Your job is to review the evidence and conclude about the reasonableness of the client's estimate.

Prior to reviewing the evidence, refresh yourself on how the estimate works by looking over the t-account and journal entries provided in Exhibit A. If this is not enough to help you remember the accounting for bad debts, stop and consult your intermediate textbook or other resources. After all, it is your job to ensure that the client's financial statements are presented in accordance with GAAP, so you must be familiar with current GAAP to perform the audit. Once you have completed your review, answer the pre-requisite questions in Exhibit A. You can check your answers by using Exhibit E to make sure your understanding is correct. You are now ready to proceed to Exhibit B to review the lead sheet, a listing of the balances under audit, and the evidence gathered.

1. Identify the evidence that supports the estimate. What facts, tests, or balances indicate that management's accrual is reasonable? That is, what evidence supports the accrual? Defend your choices.

2. Identify the evidence that contradicts the estimate. What facts, tests, or balances indicate that the accrual may not be reasonable? That is, what evidence is “disconfirming,” indicating that an audit adjustment may be needed? Defend your choices.

3. Identify the irrelevant evidence (neither supports nor contradicts the estimate). What facts, tests, or balances are irrelevant or not helpful in determining the reasonableness of the allowance for uncollectible accounts? Defend your choices.

4. Balance the persuasiveness of the supporting evidence against the persuasiveness of the contradictory evidence. Overall, rate the reasonableness of management's estimate on a scale of 0–10, with 10 = very reasonable and 0 = not reasonable at all. Defend your rating.

5. What other evidence would you gather, if any?

You have been assigned to audit a routine management estimate: the warranty reserves. The client has provided the lead sheet, and a member of your audit staff, who was pulled to work on another job, has completed the audit program and gathered considerable evidence regarding the estimate. Your job is to review the evidence and conclude about the reasonableness of the client's estimate.

Prior to reviewing the evidence, refresh yourself on how the estimate works by looking over the t-account and journal entries provided in Exhibit C. If this is not enough to help you remember the accounting for warranty reserves, stop and consult your intermediate textbook or other resources. After all, it is your job to ensure that the client's financial statements are presented in accordance with GAAP, so you must be familiar with current GAAP to perform the audit. Once you have completed your review, answer the pre-requisite questions in Exhibit C. You can check your answers by using Exhibit E to make sure your understanding is correct. You are now ready to proceed to Exhibit D to review the lead sheet, a listing of the balances under audit, and the evidence gathered.

1. Identify the evidence that supports the estimate. What facts, tests, or balances indicate that management's accrual is reasonable? That is, what evidence supports their accrual? Defend your choices.

2. Identify the evidence that contradicts the estimate. What facts, tests, or balances indicate that the accrual may not be reasonable? That is, what evidence is “disconfirming,” indicating that an audit adjustment may be needed? Defend your choices.

3. Identify the irrelevant evidence (neither supports nor contradicts the estimate). What facts, tests, or balances are irrelevant or not helpful in determining the reasonableness of the warranty reserves? Defend your choices.

4. Balance the persuasiveness of the supporting evidence against the persuasiveness of the contradictory evidence. Overall, rate the reasonableness of management's estimate on a scale of 0–10, with 10 = very reasonable and 0 = not reasonable at all. Defend your rating.

5. What other evidence would you gather, if any?

These two micro cases are designed to enhance participants' skills in evaluating management estimates and to improve their ability to identify confirming and disconfirming evidence. The learning objectives and specific experiences to prompt those skills are shown below:

A perennial auditing issue is “confirmation bias,” a tendency to notice and weight supporting evidence more readily than disconfirming evidence (McMillan and White 1993; Bamber et al. 1997). The data collected during implementation of this case supports the literature, which suggests that noticing disconfirming evidence is more difficult. This case set provides authentic audit tasks and permits students and practicing auditors to specifically work on this threat to audit quality by improving their ability to notice disconfirming evidence.

These cases require students to evaluate competing evidence that has no right answers, making the cases effective for all aspiring or practicing accountants, not just those in the audit world. Evaluating whether pieces of evidence support or do not support a theory is a foundational critical-thinking skill (Kuhn, Iordanou, Pease, and Wirkala 2008). Using evidence to test claims and create knowledge supports the development of critical thinking, a generic skill widely valued by educators and the accounting profession (Kitchener and King 1981; King and Kitchener 1994; Wolcott, Baril, Cunningham, Fordham, and St. Pierre 2002; Pathways Commission 2012).

These cases are authentic audit tasks, which provide a window to an important area of risk in the profession: auditing management estimates (PCAOB 2015). Motivational researchers suggest that asking students to address non-trivial problems in authentic projects increases task value, enhances attention needed to work at hard tasks, and fosters student thoughtfulness, all of which combine to improve student outcomes (Blumenfeld et al. 1991). As a practical matter, these cases are short, and thus reduce the potential fatigue that longer cases with overwhelming details might otherwise induce. Finally, these authentic tasks provide apprentice activities for the developing expert (Middendorf and Pace 2004).

Instructors can use these cases in the introductory auditing class in several ways, generally assigning them after the first evidence chapter (i.e., expense/payment cycle) in the course's book. For example, instructors can choose to let students work the first case quietly in class and then discuss their thoughts in groups or a whole-class discussion. After the discussion, students can attempt the second task individually or in groups, and either in class, for homework, or as an assessment. The first case takes about 40 minutes of class time. When used as an assessment in the same class period, the second case takes about another 15 minutes. For possible discussion questions and the sequence of tasks, see Exhibit F.

To extend the usefulness of the cases, students could participate in classroom discussions about how to approach the client to resolve the disagreement about the reasonableness of management's estimate. Or, instructors could form groups, assign the role of auditor or management, and ask students to resolve the disagreement and negotiate a final amount. For more advanced classes, generally the second auditing class or a graduate-level auditing class, the tasks can be used as a review or as graded assignments with discussion in between, and with the second task potentially worth a higher-point value. In addition to delivering subject matter content and serving as audit tasks, the cases require thinking and not just remembering. They also introduce ambiguity, something students tend to avoid and need coaching to accept.

These cases can also be used as a pre-test and post-test design in any audit class (see assessment pages available in Exhibits G and H). These pages are optional.

Auditing textbooks and KPMG's Judgment Framework discuss cognitive bias, including confirmation bias. As a post-mortem exercise, you could ask about which bias these cases seem to address and if this is a useful way to combat it (or what other ideas they have to reduce confirmation bias).

We suggest setting up the first case by distinguishing the auditing of “estimates” from auditing other non-estimated balances, such as market value of available-for-sale investments (which use published stock values that can be verified exactly). Ask the class to give examples of estimated and non-estimated balances and to speculate on which would be harder to audit and why. For example, estimated amounts include reserves for environmental remediation, allowance for obsolete inventory, valuation account for deferred tax assets, and so forth. Encourage a robust discussion to develop the mindset for the cases. This is a good time to introduce (if students do not suggest it on their own) how estimated amounts are often very relevant but less reliable than non-estimated amounts. This discretionary “warm up” can provide a rich review of intermediate topics.

We suggest asking students how auditors should respond when they notice disconfirming or contradictory audit evidence. Guide them into how this would cause the auditor to become more skeptical, thus “questioning contradictory audit evidence” (paragraph A24, AU-C-200). The presence of disconfirming facts increases inherent or control risks, raising the bar to reach sufficiency of evidence (the need for more confirmatory evidence to offset the contradictory signals). For example, the surge of returns at year-end increases the risk that the estimate for uncollectible accounts may be low, which prompts the need for more evidence to support the accrual.

We gave the students about ten minutes to read the case, and then quickly reviewed bad debt journal entries. We found that students appreciated the review. The overachievers enjoyed testing themselves and the others appreciated being brought current again. The review took about ten minutes, depending on the level of questions. After the GAAP refresher (and answering the questions related to the GAAP review), students evaluated and identified the evidence, either individually or in groups.

Handing out the post-case assessment (Exhibit G) is optional but can give students a bit more structure (reducing the level of difficulty). The review/coding took about another ten minutes. After everyone finished reviewing/coding, we discussed each piece of evidence and debated which column was appropriate. For examples of discussion questions, see Exhibit F. For the most part, the class eventually agreed with the suggested key, but the key was not displayed or forced on the class. Instead, students were asked to support their classification. The discussion took approximately ten minutes. In total, the first case took approximately 40 minutes of class time. For the online class, students submitted individual written responses to the first case. Alternatively, online sections might conduct a discussion using chat rooms, discussion boards, or group collaboration and then summarize back to the class.

Not all the evidence was equally convincing. For instance, a surge of returns near year-end carries more weight than following a policy. Seeing 90-day balances rise is more convincing with regard to an issue than seeing no change in 30–60 day balances. Having a new customer type may impact the accrual more than confirming balances without exception.

After the discussion on the first case had run its course, students were given the second case. Questions about warranty reserves were fielded (after the GAAP review and GAAP questions were tried and reviewed) and then students completed the second case as an assessment. Students coded each piece of evidence in the second case as either “supports” (confirming), “contradicts” (disconfirming), or “neither” (irrelevant), using the post-case assessment (Exhibit H) if desired (optional). We gave bonus points for proper classification of evidence on the second case (not just “participation”). This could have just as easily been assigned as homework.

Students had to answer an exam question at one of the three implementation schools about the case activity (why estimates were difficult to audit) and all but one student got full credit, which indicates that the discussion was interesting enough that they all got the primary “take-away” (LO B).

As with the first case, not all evidence carried equal weight. For instance, a flurry of claims at year-end would be more informative about the accrual than having a consistent policy. Finding warranty claims tucked into maintenance repairs raises more concerns than warranty expense being at the same level as last year. In other words, the contradictory evidence serves as a “red flag” that must be offset to be accepted for the evidence to be sufficient to support the estimate.

Audit firms can utilize this set of cases in new auditor training or in the firm's annual training. The cases can be tailored for use with auditors at the same level of experience or with a mixed class of auditors. For example, the warm-up and review of GAAP for the estimates can be minimized or given as pre-work, depending on the level of auditor.

During field testing of the cases, the kick-off question after the auditors completed the first case was to ask “How likely is it that all of you evaluated this evidence in similar ways?” This sparked a discussion about audit quality. The majority of the group indicated that they thought the case responses would be diverse, which stemmed from variations in skill and levels of skepticism. The auditors had a lively exchange about the first case with little involvement by the moderator (the first author). We found that auditors spent about the same amount of time completing the cases as students, but were more active and engaged during the post-case discussion.

In a de-briefing meeting following our field testing, the firm's partner in charge indicated that these materials were attractive for both identifying areas for improvement and as continuing professional development to build skills. He found the case discussions helpful and fruitful, which could, potentially, lead to more uniform expectations and skill levels across the audit teams.

Students at three universities in the southeastern U.S. completed the case set in undergraduate auditing classes taught by experienced tenure-track faculty, with some students in a face-to-face (F2F) class and the others in an online class.³ All but two sections of students (whose instructor did not get the survey in time) filled out a post-case survey (Table 1). As is typical for audit professionals, students felt more confident in finding confirming rather than disconfirming evidence. Students believed they understood the difficulty of auditing management estimates better after the casework. Students indicated that they would like more exercises like these in their audit class, even though they admitted it was “very difficult for me to do” and “made me think hard.”

Survey responses differed between F2F and online for three questions. Students in the F2F section felt more strongly that the cases illustrated audit issues better than their text. This may be a result of the strength of the book selected for the online section (one F2F section used a different book), and/or may suggest that F2F students prefer active learning to text reading. The online students were more emphatic that the cases made them think hard and were a bit more likely to indicate that the cases were over their heads, likely because the assignment was all individual work with no opportunity to collaborate or discuss with classmates.

A regional audit firm in the southeastern U.S. used these cases as part of its annual auditor training to increase audit knowledge and skill in evaluating management estimates. Post-case surveys from the 53 practicing auditors, ranging from staff to partner, are included in Table 1, Panel B, with the numbering matching the students' survey where relevant. The auditors (from staff to partner) who completed the cases reported that they were practical (92.4 percent), motivating (60.4 percent), and caused them to consider other estimates that may be difficult to audit (75.5 percent). In general, the auditors' assessments of the cases' effectiveness were similar to the students' responses. Additionally, the auditors reported that the cases promote skeptical thinking by motivating a questioning mindset about estimates in other areas.

We tracked the impact of these cases in four ways: (1) qualitatively, using a survey of participant impressions, (2) an accuracy score comparing how many pieces of evidence out of the total possible (expressed as a percent) were correctly classified on the first case compared to the second case (according to the key provided in the Teaching Notes), (3) analysis of the overall reasonableness score on the first case compared to the second case, and (4) shift in the degree of confirmation bias from case 1 or case 2.

These micro cases were first piloted in audit courses at three universities in the southeastern U.S. (n = 51), providing baseline data on how students performed without instruction. Two of the three universities subsequently implemented the cases using the instruction-then-assessment strategy (using the allowance for uncollectible accounts case for instruction and the warranty reserve case for assessment) to determine whether the case discussion provided in the first case resulted in performance gains.

We compared the assessment on the warranty reserve case after instruction (n = 23) to the student scores in the prior years. The number of warranty reserve case facts that were correctly classified was higher for students who received instruction on the first case compared to uninstructed students (F = 8.293, p = 0.005). For confirming case facts, there was a marginal difference between instructed (mean 4.33) and uninstructed students (mean 3.67) (F = 2.653, p = 0.108). In comparison, students who received instruction were significantly better (F = 7.653, p = 0.007) at identifying disconfirming evidence (mean 3.17 versus 2.10 for uninstructed). All students (instructed and uninstructed), on average, rated management's accruals as more reasonable than did the auditors, averaging 5.7 and 5.6 on case 1 and case 2, respectively. Students were less likely to update their view of the reasonableness of management's estimate after noticing more disconfirming evidence, showing their lack of experience at evaluating the persuasiveness of audit evidence.

Results by school for instructed and uninstructed students are presented in Table 2. The data show differences between schools in both the baseline and post-instruction levels, but the pattern of improvement, especially with disconfirming evidence, seems to persist across different student bodies.

The degree of confirmation bias was measured by the count of supporting evidence cited less the count of disconfirming evidence cited (with a positive amount indicating confirmation bias since the number of pieces of evidence was relatively balanced on both sides). Students' average confirmation bias dropped from a score of 1.75 on case 1 to 1.46 on case 2, indicating confirmation bias on both but to a lesser extent on the second one.

During their annual auditing and accounting update training, the first author conducted training using the first case as the pre-instruction measure and the content for discussion and feedback. The second case provided a post-instruction measure of the shift in audit skill.

The participants' attributes and pre-scores for confirming and disconfirming evidence, by position, are shown in Table 3. Not surprisingly, the managers and partners were more likely to be certified and have fraud experience. Partners were significantly more able to identify disconfirming evidence prior to the instruction. The pre-instruction levels for audit staff were similar to the pre-instruction levels for students.

While the pre- and post-scores for confirming evidence did not change significantly (t = −1.388, p = 0.171), there was a big change in the ability to identify disconfirming evidence (t = −2.637, p = 0.011). Further, the benefit was distributed across all auditor levels, as seen in Table 3, except for staff level where power was too low to detect due to small sample size (n = 7). The table also shows the drop in the overall reasonableness scores of management's estimates, a natural result of attending to the disconfirming evidence more fully.

Like with the students, the degree of confirmation bias was measured by the count of supporting evidence cited less the count of disconfirming evidence cited (with a positive amount indicating confirmation bias since the number of pieces of evidence was relatively balanced on both sides). Auditors' confirmation bias decreased from an average score of 1.16 on case 1 to −0.21 on case 2, indicating a shift away from confirmation bias. These data show that auditors had confirmation bias at the baseline measure (case 1), and, with coaching, on average lost that bias (average below 0).

This shift indicates the importance of skill-enhancing exercises such as these cases in battling against confirmation bias and improving audit quality. Importantly, the skill shifts were not confined to the least experienced staff, who clearly need more experience with evaluating evidence. These findings confirm the PCAOB's concern that most auditors could use more practice at attending to disconfirming evidence, and our findings emphasize the importance of knowledge and skill-enhancing activities such as these cases.

Our data cannot reveal the precise cause of the performance gains. That is, did practice with critical thinking, coaching on being skeptical, feedback and a second chance, more experience with evaluating evidence cumulatively, or something else lead to the improvement? While the data on uninstructed versus instructed students signal that repetition was likely important, future work could tease out whether gains might be from other case aspects and what is actually driving the gains.

This paper provides evidence that skill levels in finding disconfirming evidence are low to modest initially and can be increased across nearly all levels of auditors. In a unique contribution, these two micro cases offer both a way to assess and then a way to remedy less-than-ideal levels of skill. While there are non-skill-related reasons for weak audit judgment, identifying a fixable reason that can increase audit quality and offer professionals practice at evaluating evidence makes an incremental contribution to the profession in an area of great interest: battling confirmation bias.

Teaching Notes and the Student Version of the Case are available only to non-student-member subscribers to Current Issues in Auditing through the American Accounting Association's electronic publications system at http://aaapubs.org/. Non-student-member subscribers should use their usernames and passwords for entry into the system where the Teaching Notes can be reviewed and printed. The “Student Version of the Case” is available as a supplemental file that is posted with the Teaching Notes. Please do not make the Teaching Notes available to students or post them on websites.

If you are a non-student-member of AAA with a subscription to Current Issues in Auditing and have any trouble accessing this material, please contact the AAA headquarters office at [email protected] or (941) 921-7747.

ciia-52222_Exhibits A-D: http://dx.doi.org/10.2308/ciia-52222.s01

ciia-52222_Exhibits E_G_H: http://dx.doi.org/10.2308/ciia-52222.s02