Haphazard Sampling: Selection Biases and the Estimation Consequences of These Biases

Our recently published study, “Haphazard Sampling: Selection Biases Induced by Control Listing Properties and the Estimation Consequences of These Biases” (Hall et al. 2012; hereafter, “our study” or “the study”), provides empirical evidence concerning the reliability of haphazard sampling as a substitute for random sampling. Haphazard sampling is a nonstatistical technique used to approximate random sampling by selecting sample items without any conscious bias and without any specific reason for including or excluding items (AICPA 2012, 31). Our study's findings indicate that the properties of haphazard samples differ substantially from those of random samples. We also show that estimates derived from haphazard samples tend to exhibit unpredictable error. In this article, we discuss the motivation for the study, reasons to expect selection bias in haphazard samples, our research method, findings, and implications for practice. The study's results contribute to the literature on nonstatistical sampling and should be of interest to audit practitioners, standard-setting bodies, and regulatory authorities.

Current audit standards, including those promulgated by the U.K. Auditing Practices Board (APB), the U.S. Auditing Standards Board (ASB), the Public Company Accounting Oversight Board (PCAOB), and the International Auditing and Assurance Standards Board (IAASB) require auditors to collect sufficient and appropriate evidence before expressing an opinion (APB 2009a; AICPA 2010; IAASB 2010; PCAOB 2011a). Evidence is appropriate when it is both relevant and reliable. Compliance with this evidentiary requirement is an essential element of professional due care and affords auditors protection if they are subjected to judicial proceedings or regulatory review.

Audit samples represent an important type of evidence used to assess the error status of accounting populations and have been a source of concern in PCAOB inspections (PCAOB 2006a, 2006b, 2007, 2008a, 2008b, 2009a, 2009b, 2009c). Official pronouncements of the APB (2009b), ASB (AICPA 2010), IAASB (2010), and PCAOB (2011b) sanction both statistical and nonstatistical sampling methods, but require that all samples be selected in a manner that can be expected to yield a representative sample (APB 2009b; AICPA 2010; IAASB 2010; PCAOB 2011b). A representative sample is one that is free from material selection bias (Federal Judicial Center 2000, 244; AICPA 2012, 167). Similarly, courts in the United States generally accept both statistical and nonstatistical sample evidence (Federal Judicial Center 2000, 234), but scrutinize them for representativeness (Federal Judicial Center 2000, 232; Federal Judicial Center 2004, 103).

As a result of its professional acceptance and lower cost, nonstatistical sampling historically has played a prominent role in audit sampling. Haphazard sampling is a nonstatistical technique commonly used to emulate random sampling. To be successful, haphazard sampling must yield: (1) independent sample selections, and (2) equal selection probability across all population elements. However, a number of sampling experts have expressed doubts that haphazard sampling is a reliable substitute for random sampling (Deming 1954; Arkin 1957; Wilburn 1984). To investigate the validity of these concerns, Hall et al. (2000, 2001) tested haphazard samples chosen directly from populations and found evidence of unequal selection likelihoods. Our study extends this line of research by testing whether the properties of haphazard samples chosen from control listings exhibit the essential properties of random samples (i.e., independence and equal probability of selection).

In haphazard sampling, no explicit selection strategy is employed. Rather, the auditor selects sample elements without following any structured technique and without any specific reason for including or excluding items. To avoid selection bias, auditors are encouraged to exercise care so that features of population elements or control listing entries do not influence sample selections (APB 2009b, §530 Appendix 4; AICPA 2012, 31). For example, in applications in which sample items are selected from a control listing, the auditor selects a page from the control listing. Then, for the chosen page, the auditor scans line entries and selects one or more sample items. This process is repeated until the desired sample size is achieved. We expect this selection process to yield samples whose properties differ from those of random samples. This expectation is derived from research findings in biology and psychology that document subconscious effort minimization and diversification behaviors, and how behavior is affected by the visual appearance of an object.

Research has established that individuals subconsciously attempt to minimize effort when performing daily tasks. This innate desire for task efficiency suggests that, when haphazard sampling is employed, population elements that are easy to locate will be selected more often than population elements that are difficult to locate. In the context of haphazardly selecting sample elements from a control listing, this research suggests that auditors will tend to begin sample selections on the first page of the control listing and proceed through the control listing in serial fashion, as this strategy minimizes effort. The net result of these behaviors is that haphazard samples will overrepresent elements appearing on the first page and, given a fixed sample size, sample selections per page will tend to decline as auditors proceed through the control listing. Also, because auditors tend to proceed through control listings in serial fashion, sample selections will not be independent, but instead will be influenced by the location of the most recent selections.

At the same time, we also expect that the tendency to select fewer items from later pages will be mitigated by another subconscious behavior. Research indicates that individuals who make multiple selections in a short time period tend to categorize the choices into similar groups or brackets, and then diversify their choices over the various groups. As applied to haphazard sampling from a control listing, we expect that auditors will categorize pages based on the similarity of their serial position in the control listing. Although this categorization process may differ by individual, we expect that most auditors will include a category corresponding to the final group of pages. As the selection process proceeds, we expect auditors to focus selection activity on the final page(s) to ensure that population elements listed at the end of the control listing are not overlooked in the selection process. The result is that selections per page will increase near the end of the control listing, but whether this increased selection rate differs from that of random sampling is uncertain.

When a visual scan is conducted, but no specific object is being sought, human visual perception has been shown to automatically analyze the field of view and briefly direct attention to each visible object. When this occurs, the distinctive characteristics of objects are recognized and noted. Objects with salient features tend to draw or capture attention. As applied to sampling from a control listing, when an auditor employing haphazard sampling scans a page, subconscious processes are likely to automatically recognize and note salient features of line entries. A practical consequence of this subconscious activity is that sample selections will tend to be influenced by the line entries' distinctive features. Line entries that draw more attention will be selected more often than line entries that draw less attention. Some features that affect attentional capture include visual crowding, luminance contrast, magnitude, and serial position.

Visual crowding refers to the process whereby an object is rendered less visible when surrounded by other objects. The effect of visual crowding is that objects with fewer surrounding neighbors attract relatively more attention. Consequently, for auditors selecting haphazard samples from control listings, line entries that are preceded and/or followed by blank lines will be more visible and tend to be overrepresented in haphazard samples.

Luminance contrast refers to the extent to which the amount of light reflected from an object is different from the light reflected from the surrounding area. For example, black text on a white background exhibits higher luminance contrast than gray text on a gray background. Research in visual perception has shown that objects with higher luminance contrast are more likely to draw attention than objects with lower luminance contrast. For auditors selecting haphazard samples from control listings in which the line entries vary in background color (e.g., green-bar paper or rainbow paper), or vary in text color, line entries will vary in luminance contrast. Those line entries exhibiting greater luminance contrast are more likely to draw attention and will tend to be overrepresented in haphazard samples.

The visual magnitude of an object is another property known to affect attentional capture. Research has documented that visually large objects are more likely to attract attention than are visually small objects. Consistent with this finding, Hall et al. (2000) found that larger population elements were overrepresented in haphazard samples. Consequently, for auditors selecting haphazard samples from control listings, line entries with larger numeric magnitudes representing monetary balances or quantities are more likely to draw the auditor's attention and, therefore, will tend to be overrepresented in haphazard samples.

Finally, the reading of English text proceeds from page top to page bottom. As a result, English-speaking auditors scanning an English language control listing are expected to scan line entries in serial fashion, starting with the first (top) line and concluding with the last (bottom) line. After scanning a page, sample selections can be expected to be influenced by those line entries that are more likely to attract attention. Numerous studies have demonstrated that items at the beginning and end of lists are more likely to attract attention. This aspect of visual perception suggests that the first few and last few lines on each page will tend to stand out and be overrepresented in haphazard samples.

To test the preceding expectations, we created two control listings representing a population of accounts receivable and a population of inventory items. The accounts receivable control listing consisted of 22 pages with 792 customer accounts, while the inventory control listing consisted of 26 pages with 1,404 inventory items. Line entries exhibited diverse visual properties (details are available in Hall et al. [2012]). We then conducted three experiments in which participants were instructed to select haphazard samples from the control listings.

Participants in the first experiment were 75 students enrolled in either senior or master's-level accounting courses at a public university located in the southwestern United States. The second experiment utilized 40 university students in the United Kingdom who were enrolled in either senior or master's-level accounting courses. The students from the United States and United Kingdom serve as effective proxies for entry-level auditors, who select most samples. The third experiment utilized 53 audit seniors from two offices of a Big 4 audit firm located in the southwestern United States. Because of time constraints, the audit seniors sampled only from the inventory control listing.

Upon completion of the sample selection process, all participants completed an exit survey to determine: (1) their commitment to the sampling task, (2) whether they used haphazard sampling, and (3) how confident they were regarding the representativeness of their samples. Finally, we analyzed the haphazard samples, by participant group, to determine if their properties matched those of random samples (i.e., independence and equal probability of selection).

Responses to the exit survey confirmed that participants were committed to selecting representative samples and that they did use haphazard sampling. Student participants expressed limited confidence in the representativeness of their samples while audit seniors, as might be expected, expressed more confidence. Despite these survey results, analyses of participants' samples disclosed multiple deviations from the properties of random samples.

As expected, we observed unequal page selection rates. Most participants began the sample selection process on the first page of control listings. Also, sample selections exhibited a high positive correlation, indicating that participants tended to proceed through the control listings in serial fashion. Statistical analyses confirmed that participants exhibited higher selection rates for early pages, followed by declining selection rates for middle pages, with an upturn in selection rates for ending pages. All of these results are inconsistent with the properties of random samples.

Line selection rates also were unequal and consistent with expectations that visual perception biases influence sample selections. Line entries with a low level of visual crowding tended to have higher selection rates than line entries with a high level of visual crowding. Similarly, line entries with a high level of luminance contrast were selected more often than line entries with lower levels of luminance contrast. All participant groups exhibited higher selection rates for line entries with larger numeric magnitudes, but statistical tests were not significant for the samples selected by audit seniors.¹ Finally, statistical tests confirmed that lines at the top and bottom of pages were overrepresented in each participant group's samples. As with page selection, these results are inconsistent with the properties of random samples.

Our study also tested whether participants' confidence in the representativeness of their samples and participants' audit experience were associated with haphazard samples that better matched the properties of random samples. Tests comparing the properties of haphazard samples selected by high-confidence and low-confidence participants disclosed that the samples selected by participants with high confidence were no closer to random samples than the samples selected by participants with low confidence. Also, comparisons of samples selected by students versus those selected by audit seniors exhibited no systematic relationship between audit experience and the ability to emulate random sampling. We posit that these results arise from the fact that auditors neither receive substantial training in haphazard sampling nor feedback regarding the biases exhibited by their haphazard samples. Both of these factors are important elements in the acquisition of expertise and the ability to apply expertise consistently.

The combined findings of Hall et al. (2012) and those of Hall et al. (2000, 2001) suggest that the properties of haphazard samples, whether chosen from control listings or from the actual population, are likely to differ from those of random samples. Subconscious effort minimization and diversification behaviors, coupled with visual perception artifacts, yield samples that violate requirements for independence and equal selection probability. These violations, in turn, are likely to produce biased error projections with difficult to discern risk properties. Although widely used and specifically identified in audit standards as a sampling technique that can be employed to obtain a representative sample, haphazard sampling may not be a reliable substitute for random sampling. As demonstrated by the infamous McKesson & Robbins case (Barr and Galpeer 1987; Bealing et al. 1996), the use of a professionally sanctioned but deficient audit procedure brings increased risk of audit failure, legal liability, and regulatory scrutiny.

In some audit circumstances, statistical methods are impractical because of cost or an inability to meet technical requirements (see, Wilburn 1984, 17; Guy et al. 1998, 150; AICPA 2012, 15). For example, statistical methods generally are not cost effective when auditing small populations. Statistical methods also may be impractical when the audit objective is to test for completeness. For these situations, despite their potential weaknesses, reliance on nonstatistical methods may be necessary. When auditors use nonstatistical techniques, they should undertake and document debiasing efforts. One debiasing procedure in current use, increasing sample size to reduce haphazard sampling selection bias, has been shown to provide a small reduction in selection bias (Hall et al. 2001). Another debiasing strategy is to avoid both: (1) always starting the selection process on the first page of control listings, and (2) proceeding through control listings in a serial fashion. Research aimed at the development of additional debiasing procedures appears warranted. Providing auditors with formal training in debiasing procedures and feedback regarding sample quality should be undertaken where feasible.

Currently, audit standard-setting bodies sanction the use of haphazard sampling but do not provide guidance for discerning when it can be expected to yield a representative sample. Guidance on this issue would be beneficial, and might include consideration of auditor training in debiasing techniques, monetary coverage provided by a census stratum of individually significant items, and prior auditor knowledge of the underlying population. Other factors that might bear upon the decision to use haphazard sampling include the feasibility of random sampling, materiality of the audit area, expected error relative to tolerable error, and acceptable sampling risk.