Audit of the Board of Bar Examiners Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Board of Bar Examiners.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Board of Bar Examiners (BBE) for the period July 1, 2016 through June 30, 2018.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question that the audit intended to answer and the conclusion we reached regarding each objective.

Objective	Conclusion
Does BBE evaluate the character and fitness of petitioners in accordance with the Rules of the Board of Bar Examiners as authorized by Section 7 of Supreme Judicial Court (SJC) Rule 3:01?	Yes
Does BBE provide applicants with testing accommodations in accordance with Americans with Disabilities Act of 1990 (ADA) requirements?	Yes

 

To achieve our objectives, we gained an understanding of the internal controls we deemed significant to our audit objectives through inquiries and observations. We evaluated the design of controls over BBE management’s review of both nonstandard testing accommodation (NTA) and character and fitness evaluations. We also evaluated the effectiveness of controls over the character and fitness evaluation process. We assessed whether these controls operated as intended during the audit period.

We performed the following procedures to obtain sufficient, appropriate audit evidence to address the audit objectives.

• We obtained from BBE’s FileNet file management system a list of the 4,534 petitions for admission to the Massachusetts bar from our audit period. We segregated them into two groups: (1) petition by examination and (2) petition by motion or Uniform Bar Examination (UBE) transfer.
• For petitions by examination, we selected a statistical random sample using a 90% confidence level and a tolerable error rate of 10%. We sampled 24 petitions from the 3,961 total petitions by examination from the audit period. To determine whether a petitioner’s character and fitness were evaluated in accordance with Section 7 of SJC Rule 3:01, we inspected the following: copies of the petition, law school certification, letters of recommendation (a minimum of two), issues identified in the documentation and addressed by the petitioner, and actions taken by a staff investigator to resolve the issues.
• Using a nonstatistical sampling method, we selected a random sample of 30 petitions from the 573 total petitions by motion or UBE transfer from the audit period. We acquired the 30 petitions with the assistance of BBE’s staff. To determine whether a petitioner’s character and fitness were evaluated in accordance with Section 7 of SJC Rule 3:01, we inspected the following: copies of the petition, law school certification, letters of recommendation (minimum of three), bar certification and letter of good standing from another jurisdiction, grievance letters (if applicable), background reports, issues identified and addressed by the petitioner, and actions taken by a staff investigator to resolve the issues.

We inspected BBE’s list of petitioners who applied for NTAs during the audit period; this list is generated and managed separately from the list of petitions to the bar. We selected a nonstatistical random sample of 35 out of 194 NTA applications from the audit period. We inspected original source documents for completeness, descriptions of the applicants’ self-identified specific disabilities, accommodation histories, medical histories and supporting documentation, academic transcripts, and external medical/clinical consultation reports (if requested) to ensure that NTAs were provided in accordance with ADA requirements.

Whenever nonstatistical sampling was used, we could not project our result to the entire population.

Data Reliability

We determined the reliability of the FileNet data by performing interviews and testing information technology controls over user identification and authentication policies and procedures, account management, security training, and monitoring configuration. We ensured the completeness of our list of petitions by judgmentally selecting 15 petitions from a BBE file cabinet and verifying that they corresponded to the exam dates in FileNet. In addition, we judgmentally selected 25 petitions from the FileNet list and traced the bar petition numbers to the hardcopy petitions. Further, we analyzed the list for missing data, hidden data and columns, duplicate information, and active filters. We determined that the data from FileNet were sufficiently reliable for the purposes of this audit.

We determined the reliability of the NTA list by performing interviews with BBE management. We ensured the completeness and accuracy of the list by judgmentally selecting 10 applications from the file cabinet and verifying that they were included on the list of applications in FileNet. Further, we judgmentally selected 10 applications from the NTA list and traced the bar petition numbers to the NTA applications. We then analyzed the list of applications for missing data, hidden data and columns, duplicate information, and active filters. We determined that the data from the NTA list were sufficiently reliable for the purposes of this audit.

Date published:	April 26, 2019