This article summarizes “The Impact of Benchmark Set Composition on Auditors' Level 3 Fair Value Judgments” by Bhattacharjee, Moreno, and Wright (2019), which examines how auditors' judgments of the reasonableness of a client's discount rate for a Level 3 investment are impacted by client-provided benchmarks. In two experiments, the authors find that audit seniors' and managers' judgments of a client-preferred discount rate for an investment are inappropriately influenced by the set of peer companies provided by the client as justification. Managers are less susceptible than seniors, likely due to highly developed knowledge structures. Results suggest that providing structured audit guidance to the seniors for conducting analyses somewhat reduces but does not eliminate this effect. The study's findings have implications for other auditing contexts using benchmarking such as goodwill impairment, inventory obsolescence, and valuation estimates and for audit firms when auditing complex estimates and determining staffing of audit engagements.

Auditors frequently use benchmarking analyses to evaluate the appropriateness of a company's estimates or assumptions underlying those estimates (Acuitas, Inc. 2017; PCAOB 2014, 2016). Specifically, estimates such as Level 3 investments, goodwill impairment, inventory obsolescence, and valuation estimates, require management to provide subjective inputs and assumptions which may be in the form of a benchmark set. However, company managers can strategically select benchmark data. For example, public companies “cherry-pick” peers to justify inflated executive pay to shareholders, masking overpayment to executives (Melin and Diamond 2015). As such, when evaluating the appropriateness of estimates, auditors may be influenced by the choice that company managers make when selecting benchmark data. The International Forum of Independent Audit Regulators (IFIAR) found that 29 percent of all PCAOB inspection findings were related to estimates, most of which “related to failure to assess the reasonableness of assumptions, including consideration of contrary or inconsistent evidence” (Farr 2019). The PCAOB and the SEC have specifically noted concerns with auditors' assessments of benchmark data when evaluating inputs of significant assumptions for estimates (Acuitas, Inc. 2017).

A recent study, “The Impact of Benchmark Set Composition on Auditors' Level 3 Fair Value Judgments” (Bhattacharjee, Moreno, and Wright 2019), predicts and finds that audit seniors' judgments of the reasonableness of a client-preferred discount rate for a Level 3 investment are inappropriately influenced by the set of peer companies provided by the client as justification. Rather than independently comparing each peer company to the investment to determine an appropriate benchmark, the authors find that senior auditors in the first experiment inappropriately compared the attributes of the peer companies to the client-provided set of peers. The presence of a peer company with low similarity to the investment made a moderately similar peer with an aggressive discount rate appear more similar to the investment leading auditors to inappropriately accept a more aggressive discount rate. Moreover, the first experiment also finds that providing very structured audit guidance for conducting benchmarking analysis can be somewhat successful in reducing, but not eliminating, this effect. In a second experiment, the authors find that audit managers are less susceptible to these contrast effects as compared to audit seniors. While not surprising, the results from the second experiment suggest that audit managers who typically have five to eight years of audit experience demonstrate less biased judgments in this benchmarking task perhaps due to their more developed knowledge structures than audit seniors who typically have two to four years of experience. Given the widespread use of benchmark data, this study has significant implications for audit firms and standard setters as they try to deal with deficiencies in auditors' judgments when auditing complex estimates.

PCAOB inspection reports have specifically identified deficiencies in auditors' evaluations of the comparability of peer companies that management uses to support their estimates. The reports note that auditors sometimes fail to fully evaluate whether the provided peer companies are comparable to the client. Bhattacharjee et al. (2019) examine whether the composition of the benchmark set provided by the client to support the assumptions underlying the estimates impacts auditors' assessments of the appropriateness of the estimate.

Consider a fair value scenario in which management compares the discount rate for an investment to two peer companies to support their preferred discount rate. One peer company may have a more conservative rate than the client-preferred rate and have superior similarity to the Level 3 investment as it is in the same broad industry and has similar product lines. Another peer company may have a more aggressive rate than the rate of the first peer company but may be moderately similar to the Level 3 investment as it is in the same broad industry but the product lines are in a different subsector. In this situation, auditors are likely to identify that the moderately similar peer company is an inferior comparison to the investment relative to the peer company with superior similarity and will likely recommend a conservative rate.1

However, to further support their discount rate, a client could add a third peer company that has low similarity to the investment as its industry is only tangentially related to the investment. Contrast effects research would suggest that because the attributes of the third peer company have low similarity to the investment, the moderately similar peer company with an aggressive discount rate will now appear more similar to the investment. Contrast effects occur when an alternative is rated more positively when compared (contrasted) to a less positive alternative than when it is compared to a more positive alternative (Huber, Payne, and Puto 1982). Rather than independently comparing each peer company to the investment to determine an appropriate benchmark, research on contrast effects suggests that auditors may also contrast the low similarity peer company to the moderately similar peer which will inappropriately impact auditors' evaluation of the moderately similar peer. Psychology research has shown that a decision maker's choice of an alternative in a set changes relative to the composition of the other alternatives in the set (e.g., Bhatia 2013; Huber et al. 1982; Noguchi and Stewart 2014). This research finds that preferences depend on the comparisons that individuals make among the attributes of the alternatives in the choice set, in addition to the attributes of the alternatives themselves. Based on contrast effects research, Bhattacharjee et al. (2019) predict that because the attributes of the third peer company have low similarity to the investment, the moderately similar peer with an aggressive discount rate might appear more similar to the investment leading auditors to inappropriately accept a more aggressive discount rate.

Psychology research also suggests that the key to overcoming contrast effects is to get individuals to assess attribute values based on the values themselves and not based on a comparative process (e.g., Palmer, Maurer, and Feldman 2002; Carlson and Bond 2006; Noguchi and Stewart 2014). Thus, Bhattacharjee et al. (2019) also examine if structured guidance that explicitly instructs auditors to compare the attributes of the peer companies to the actual investment when conducting benchmarking analysis may be effective at reducing contrast effects. Finally, prior research suggests that knowledge structures developed through experience sometimes weaken the impact of contrast effects on professionals' judgments since the knowledge structures allow experienced decision makers to focus on relevant information (e.g., O'Reilly, Leitch, and Wedell 2004). Thus, the current study separately examines if contrast effects impact audit managers' decisions during the benchmarking task in a second experiment.

One hundred eleven auditors with an average of 36.6 months of experience participated in Experiment 1. Auditors were tasked with assessing the discount rate assumption of a Level 3 investment of a hypothetical client. Auditors were told that management utilized the discounted cash flows method to assess the fair value of the investment. Management provided a memo to support their 11 percent discount rate. The memo outlined the risk profile of the Level 3 investment using four attributes of the company—the industry (biomedical technology industry), age (established in 2011), number of product lines (one product line), and size ($8.4 million total assets). The client also provided attribute details of different peer companies. The peer companies were used to manipulate the benchmark set composition based on their degree of similarity to the four attributes of the Level 3 investment.

As presented in Table 1, auditors were divided into six experimental conditions. Three experimental conditions manipulated the benchmark set composition: Two-Peers condition, Plausible-Third-Peer condition, and Implausible-Third-Peer condition. The Two-Peers condition, a control condition, included two peer companies; one peer with superior similarity and a conservative discount rate of 14.5 percent and another peer with moderate similarity and a more aggressive discount rate of 10.5 percent. The Plausible-Third-Peer condition included an additional peer company with lower similarity to the Level 3 investment and a 9 percent discount rate which was close but more aggressive than the moderate peer company. This peer company was designed to make the moderate peer company appear more similar to the Level 3 investment due to contrast effects. The Implausible-Third-Peer condition was designed to rule out the possibility that the results were only due to the presence of a third peer company rather than contrast effects based on the composition of the benchmark set. This condition included a third peer company that had such low similarity to the Level 3 investment that auditors were not expected to use it in their judgment. To explore the role of structured guidance on auditors' judgments, three additional groups based on this peer company manipulation received structured guidance. For the structured guidance conditions, auditors were told to assess the discount rate assumption by comparing the peer companies to the investment. Auditors were given explicit instructions to focus on the attributes of the investment relative to the peer companies.

TABLE 1

Investment and Peer Company Details and Overview of Benchmark Set Composition of Experimental Conditions

Investment and Peer Company Details and Overview of Benchmark Set Composition of Experimental Conditions
Investment and Peer Company Details and Overview of Benchmark Set Composition of Experimental Conditions

The results, depicted graphically in Figure 1, suggest that auditors were significantly influenced by the benchmark set composition. Sixty-five percent of auditors in the Plausible-Third-Peer condition assessed that the discount rate should be less than or equal to the 11 percent client-preferred rate as compared to 5.9 percent in the Two-Peers condition. This difference translates into a significant difference in the mean unrealized gain assessments between the Two-Peers condition ($3,273,213) and the Plausible-Third-Peer condition ($8,259,488). Since only 9.5 percent of the auditors in the Implausible-Third-Peer condition assessed the discount rate to be less than or equal to 11 percent, the results do not appear to be due to the mere presence of a third benchmark.

The results also suggest that structured guidance can help partly reduce contrast effects from the benchmark set composition. There were significant differences between the percentage of auditors who assessed that the discount rate should be less than or equal to the client preferred rate in the Plausible-Third-Peer condition when structured guidance was given (28.6 percent), than in the Two-Peers condition (0). However, there was a significant difference in the Plausible-Third-Peer conditions with and without structured guidance (28.6 percent versus 65 percent), suggesting that structured guidance had some impact in reducing contrast effects.

Experiment 2 was conducted to determine if the results were robust to auditors with higher experience levels. While audit managers have highly developed knowledge structures, contrast effects have been shown to be quite robust within and outside accounting (e.g., Palmer and Gore 2014; Shelton 1999; Hammersley 2006; O'Reilly et al. 2004). Thirty-three managers with an average of 77.52 months of audit experience participated in Experiment 2. Audit managers were randomly assigned to two conditions: the Two-Peers condition and the Plausible-Third-Peer condition. Experimental materials were identical to Experiment 1.

As depicted graphically in Figure 2, 25 percent of managers in the Plausible-Third-Peer condition assessed that the discount rate should be less than or equal to the 11 percent client-preferred rate as compared to 0 of the managers in the Two-Peers condition. Thus, contrast effects still exist for managers. With caution, the authors examined if the discount rate pattern was the same between the seniors in Experiment 1 and managers in Experiment 2. A lower percentage of managers in the Plausible-Third-Peer condition (25 percent) agreed with the client than seniors in the Plausible-Third-Peer condition (65 percent). Subject to the caveat of making comparisons across experiments, managers seem less susceptible to contrast effects.

While the use of benchmarking can be a useful tool for auditors to evaluate the assumptions underlying Level 3 investments, the results of Bhattacharjee et al. (2019) suggest auditors may fall victim to the impacts of contrast effects. Small differences in discount rates can have material impacts on a client's financial statements. Understanding how to improve auditors' judgments in these subjective areas remains an important area for practice.

The results of Experiment 1 suggest that when senior auditors are provided structured guidance about performing comparisons, contrast effects are less. Specifically, when directed to compare the attributes of the client (e.g., industry, age, total assets, revenues, number of product lines) to those of the benchmarked peers, comparisons of discount rates are more appropriate, and the impact of contrast effects is reduced. Structured guidance helps less experienced auditors focus on relevant information and ignore irrelevant information. Using structured guidance may be used to assess the reasonableness of other types of estimates, like discount rates used to calculate pension liabilities or asset impairments.

Results from Experiment 2 confirm that more experienced auditors are less susceptible to contrast effects. Well-developed knowledge structures may enable more experienced auditors to have a better focus on more important, relevant information. While a subject of future research, additional reductions in managers' susceptibility to contrast effects may be attained by changing the order in which the peer companies within the benchmark set are presented or through a form of structured guidance when reviewing work performed by staff auditors. These results have implications for engagement staffing and audit task assignment.

While not addressed in the current study, auditors in practice likely consider the completeness of the benchmark data set that a client provides. The addition of peer companies to benchmark sets may reduce the risk that the auditor will be unduly persuaded by the client-provided data. Unfortunately, when auditing Level 3 investments, the population of comparable companies is likely to be small.

Acuitas, Inc.
2017
.
Acuitas, Inc.'s survey of fair value audit deficiencies
.
Bhatia,
S.
2013
.
Associations and the accumulation of preference
.
Psychological Review
120
(
3
):
522
543
.
Bhattacharjee,
S.,
Moreno
K. K.,
and
Wright
N. S.
2019
.
The impact of benchmark set composition on auditors' Level 3 Fair Value Judgments
.
The Accounting Review
94
(
6
):
91
108
.
Carlson,
K.,
and
Bond
S.
2006
.
Improving preference assessment: Limiting the effect of context through pre-exposure to attribute levels
.
Management Science
52
(
3
):
410
421
.
Farr,
L.
2019
.
A refined approach to auditing accounting estimates. Journal of Accountancy. Available at: https://www.journalofaccountancy.com/news/2019/jun/auditing-accounting-estimates-201921413.html
Hammersley,
J. S.
2006
.
Pattern identification and industry-specialist auditors
.
The Accounting Review
81
(
2
):
309
336
.
Huber,
J.,
Payne
J. W.,
and
Puto
C.
1982
.
Adding asymmetrically dominated alternatives: Violations of regularity and the similarity hypothesis
.
The Journal of Consumer Research
9
(
1
):
90
98
.
Melin,
A.,
and
Diamond
J. S.
2015
.
How companies justify big pay raises for CEOs. Bloomberg.com (June 4). Available at: https://www.bloomberg.com/graphics/2015-executive-pay-peer-groups/
Noguchi,
T.,
and
Stewart
N.
2014
.
In the attraction, compromise, and similarity effects, alternatives are repeatedly compared in pairs on single dimensions
.
Cognition
132
(
1
):
44
56
.
O'Reilly,
D.,
Leitch
R.,
and
Wedell
D.
2004
.
The effect of immediate context on auditors' judgments of loan quality
.
Auditing: A Journal of Practice & Theory
23
(
1
):
89
105
.
Palmer,
J. K.,
and
Gore
J. S.
2014
.
A theory of contrast effects in performance appraisal and social cognitive judgments
.
Psychological Studies
59
(
4
):
323
336
.
Palmer,
J. K.,
Maurer
T. J.,
and
Feldman
J. M.
2002
.
Context and prior impression effects on attention, judgment standards, and ratings: Contrast effects revisited
.
Journal of Applied Social Psychology
32
(
12
):
2575
2597
.
Public Company Accounting Oversight Board (PCAOB).
2014
.
Report on 2013 Inspection of KPMG LLP. PCAOB Release No. 104-2014-167
.
Washington, DC
:
PCAOB
.
Public Company Accounting Oversight Board (PCAOB).
2016
.
2015 Inspection of MaloneBailey, LLP. PCAOB Release No. 104-2016-163
.
Washington, DC
:
PCAOB
.
Shelton,
S. W.
1999
.
The effect of experience on the use of irrelevant evidence in auditor judgment
.
The Accounting Review
74
(
2
):
217
224
.
1

This benchmarking analysis procedure is only one input in an auditor's judgment of the reasonableness of the discount rate. The auditor will also consider other inputs (e.g., economic and client risk factors) to assess the client's discount rate. However, the summarized study's focus is on benchmarking analysis and holds all other inputs constant across experimental conditions.