Practitioner Summary of Detection and Severity Classification of Sarbanes-Oxley Section 404 Internal Control Deficiencies Free

Our recently published study, “Detection and Severity Classification of Sarbanes-Oxley Section 404 Internal Control Deficiencies” (Bedard and Graham 2011), provides unique research evidence on a highly important issue in current auditing practice—the contribution of auditor assessment and testing of internal controls over financial reporting to the Sarbanes-Oxley Section 404 process.¹ Specifically, we use proprietary data provided by several large auditing firms, obtained through confidentiality agreements, to investigate the number and nature of internal control deficiencies (ICDs) detected, who (client or auditor) detected them, how they were detected, and factors associated with their severity, as assessed by the client and/or auditor. To our knowledge, this research is unique in that it is the only study based on data that includes the entire set of company control deficiencies detected in Section 404 engagements, and information on detection processes. By identifying the independent auditor's contributions to the Section 404 process, our study speaks to the value of auditor testing of and reporting on internal controls under Section 404(b), a key component of the Act's provisions to improve financial reporting quality.²

This research is important because the requirements of this Act have been under fire since it was passed by Congress and signed into law in 2002.³ Both academic research and the financial press have documented the high cost of auditor testing, and intense lobbying by corporations is intended to prevent its extension to non-accelerated filers. After years of postponement, Congress exempted non-accelerated filers from obtaining an auditor's report on their internal control effectiveness (Section 404(b)) in the Dodd-Frank Financial Reform Act of 2010 (U.S. Congress 2010). Further, even the application of Section 404(b) for accelerated filers remains under threat, as the U.S. remains one of the few countries to require that auditors assess, test, and report on the effectiveness of internal controls over financial reporting.⁴

Our results show that only about 4 percent of detected ICDs in the sample were designated as material weaknesses, implying that the publicly available Section 404 reports only identify the “tip of the iceberg” in terms of the detected control flaws that may affect financial reporting quality. The study's findings also affirm the importance of auditor testing of internal controls over financial reporting, as auditors detect about three times as many control deficiencies as client personnel, even though client personnel perform tests before auditors. Further, when clients provide a preliminary classification of the severity of control flaws that they detect, they tend to underestimate the likelihood that the deficiencies might lead to material misstatements. It is important to note that these insights would not have been possible without the cooperation we received from multiple firms. This cooperation enables us to examine a larger sample of ICDs than a single firm could reasonably provide, and improves our ability to generalize our findings beyond a single firm's experience.

Our sample includes 76 audit engagements (44 companies) in 2004 and 2005. Participating firms randomly selected engagements that fit our sample criteria (smaller accelerated filers with revenues less than a billion dollars, in non-regulated industries). Engagement personnel completed a spreadsheet with information about the client business (e.g., industry, values of some financial accounts, judgments about quality of information technology support, and internal audit quality) and the specific control flaws detected (e.g., who detected them and how they were detected).⁵ The mean total assets of the sample companies are $502.5 million and the mean ratio of liabilities to assets is 0.4. Sample companies have on average 2.5 segments and 4.9 locations. About 30 percent of the companies have been public for fewer than five years, and about half are manufacturers.

Among the engagements in the sample, 15 companies (20 percent) have at least one material weakness (MW) that was unremediated as of year-end, resulting in an audit opinion noting ineffective internal controls.⁶ In addition to the 15 companies with MWs, another 39 have at least one significant deficiency (SD). This implies that only 22 companies (29 percent) had no ICDs sufficiently severe to require reporting, at least to the audit committee. At the individual control level, clients and auditors identified 3,990 ICDs in total. Of those, 4 percent are MWs, 12 percent are significant deficiencies, 58 percent are control deficiencies (the least severe of the ratings), and 26 percent were remediated prior to the balance sheet date.⁷

Table 1 provides characteristics of sample companies relevant to the status of internal controls and management of the Section 404(a) process, comparing companies with effective controls to those with ineffective controls. Auditors assessed control reliance as “strong” or “maximum” in 49 percent of sample companies, with more control reliance for companies with no MWs. The mean assessment of the client's effectiveness in integrating technology into the Section 404(a) process is 3.3 on a scale of 1 (highly ineffective) to 5 (highly effective). These findings suggest that companies with effective controls have better perceived controls at the outset and better IT integration. Companies began their 404 projects an average of six months before year-end and three-fourths of companies engaged an outside consultant in their Section 404(a) project.⁸ Only 46 percent of the companies had an existing internal audit function. Most of the company projects reported the results of assessments and testing controls through management, as only 34 percent indicated that they reported independently to the audit committee.

Our data also contain information from auditors about how the ICDs were detected. Under the directed SOX process, companies assess their own control design issues and test their controls before the audit process. Results reported in Table 2 show that companies, in performing their required assessment, detected surprisingly few ICDs. Auditors detected 72 percent of all deficiencies and 84 percent of the MWs. Detecting deficiencies is the first step toward improving the quality of internal controls, and our results imply that, without auditor involvement, most of the control deficiencies would not have been recognized. The statistic that most of the ICDs (64 percent) were identified through control tests reinforces the value of controls assessment and testing. This observation holds true for the detection of MWs, where 59 percent were identified through control tests. If the presence of misstatements alone were used as a detection device, our results imply that fewer than 30 percent of MWs and 12 percent of all deficiencies would have been identified.

In addition to providing descriptive information on how ICDs were detected, another main objective of our study was to highlight characteristics of companies and their ICDs associated with greater severity. The full paper reports two regression models whose results we summarize in Table 3; specifically, we show the factors that are statistically significant, and their directions. Model 1 compares MWs and SDs to other control deficiencies. This model examines an important threshold, as all SDs and MWs are required to be reported to the audit committee, and SDs carry an expectation of near-term correction. A failure to identify one or more SDs or MWs, when they do exist, increases the risk of incorrect assessments of control effectiveness and the risk of future misstatements. Model 2 compares SDs to MWs, specifically the factors associated with the materiality of the more severe deficiencies. Both models contain four categories of variables, shown as sections in Table 3: (A) client detection processes; (B) auditor detection processes; (C) specific types of entity-level (general) control deficiencies; and (D) specific types of account-specific control deficiencies.

Table 3 reveals that a number of factors are associated with greater severity of identified ICDs. Regarding client detection factors (Section A), we find that clients with greater independence in the Section 404(a) process (i.e., no filtering of results of testing through management before reporting to the auditor or audit committee) report more severe ICDs. We also find that clients using a large auditing firm as a consultant (an indication of process quality) report more severe control problems. We interpret both results to mean that a stronger client process results in more accurate detection of control flaws.⁹ The presence of an internal audit function is not found to affect severity of control flaws. However, clients that more effectively integrate IT personnel into their Section 404(a) processes have less severe control problems, implying stronger underlying IT controls in those companies. Prior auditor expectations to rely on controls as an audit strategy also are associated with less severe ICDs.

Results regarding auditor processes (Section B) show that auditor detection is associated with higher severity, as is the amount of lead-time that auditors have built into their assessment, the presence of a misstatement, and detection by substantive testing. While these results may not be surprising, Table 2 shows that many ICDs without an associated misstatement also are classified as severe, and many ICDs are detected through controls design assessments and control tests, not through substantive tests of account balances and transactions. We believe that the evidence of a misstatement is likely important in supporting a more severe control assessment in client discussions, even though the definitions of SDs and MWs are not dependent on finding a current misstatement.¹⁰

Section C shows the impact of “entity-level” controls on severity; our regression models include variables representing the five COSO components (COSO 1992, 2006). Model 2 shows that Control Environment ICDs are more likely to be MWs. Entity-level controls in the general ledger and information technology general controls also are more severe in Model 1. In contrast, COSO Monitoring, Information/Communication, and Risk Assessment are less likely to be severe. A plausible reason for this result is that deficiencies related to these COSO categories were not well understood relative to control deficiencies identified in the Control Environment and account and transaction stream controls (Control Activities), which were established in the professional standards discussions even prior to the 1992 Framework (COSO 1992).¹¹

The Section D results concerning specific accounts reveal that revenue recognition issues are associated with higher severity ratings, consistent with concerns expressed by the SEC (1999) in the period just before SOX. Tax issues also are associated with higher severity assessments, which we expected because, under the then-recent SEC independence rules, some companies in this time period became responsible for the first time for performing the year-end tax accrual task. Additionally, the tax accrual, by its nature, is performed at or after year-end, thereby reducing or eliminating the chance for remediation.

We also asked auditors to identify whether the client made a preliminary severity classification of the ICDs that they detected. Comparing these client classifications to those of the engagement teams shows that companies do not robustly assess ICD severity. Of the 28 ICDs detected by clients that the auditor classified as MWs, 20 (71 percent) were initially considered only SDs or deficiencies by clients. In addition, of the 91 SDs detected by clients, 59 (65 percent) were initially considered only deficiencies by clients. These results imply that, without auditor input regarding the implications of control flaws for financial reporting, most client-detected ICDs would not be reported to the public, audit committee, or top management.

Our research examines ICDs detected in a sample of public companies, along with detection methods and factors associated with auditor assessments of severity. This study is important because research using only publicly available data on SOX effectiveness is limited in two ways. First, only MWs not remediated by year-end are visible and, second, ICDs of lesser severity than MWs are never visible, whether remediated or not. Our results show that the publicly reported (visible) MWs are only about 4 percent of total ICDs detected in the sample companies' SOX processes. One implication is that many corporations, even those with clean Section 404 opinions, potentially have a large number of important control deficiencies that are not visible to the public. As a result, publicly disclosed MWs may not be representative of all or even just severe (MWs and SDs) deficiencies. Thus, public disclosures may not provide a good surrogate to use when analyzing the characteristics of ICDs or the impact of SOX on improvements in corporate internal controls.

We also show that companies failed to detect 72 percent of the all ICDs that were eventually identified, and failed to detect 84 percent of the MWs. An important implication is that, in the absence of clear improvements in the company processes that were applied in this period, the annual Section 404(a) self-assessment now required of smaller public companies, or the quarterly Section 302 assessment required of all public companies, may not be effective in detecting and correcting many existing internal control deficiencies. Further, companies in our sample tended to under-classify the severity of the deficiencies they did identify. Most notably, they did not recognize 71 percent of MWs as deficiencies at that level of severity. Under current requirements, non-accelerated filers must self-assess and report on the effectiveness of their internal controls. Again, the implications of these findings for a solely company-driven process like Section 302 and 404(a) are obvious, calling into question the wisdom of the Dodd-Frank Act's suspension of Section 404(b) auditor reporting on internal controls for non-accelerated filers. In sum, our study provides clear evidence of some of the benefits of auditor involvement in Section 404 reporting. In so doing, it informs a public policy debate that has been unbalanced in its predominant focus on costs.