Audit of the Massachusetts Board of Bar Examiners Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Massachusetts Board of Bar Examiners.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Massachusetts Board of Bar Examiners (MBBE) for the period March 1, 2020 through November 30, 2021.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Did MBBE administer the Uniform Bar Examination (UBE) remotely in October 2020 and February and July 2021, as required by Superior Court Standing Order 8-20?	Yes
Did all MBBE employees and board members complete the conflict of interest training during the audit period, as required by Section 28 of Chapter 268A of the General Laws?	No, see Finding 1

 

To achieve our audit objectives, we gained an understanding of the internal control environment we determined to be relevant to the objectives by conducting inquiries with MBBE officials and reviewing MBBE’s internal control plan and applicable agency policies and procedures.

To obtain sufficient, appropriate audit evidence to address our audit objectives, we performed the following procedures.

To determine whether MBBE administered the UBE remotely during the audit period, we obtained a list of all applicants who took the UBE during the audit period from the Case Management System (CMS), which holds applicants’ information and is maintained by the Supreme Judicial Court (SJC). For each unique case number, we determined whether the applicant associated with that case number had a final score of greater than zero and had an entry date (application submittal date) before the scheduled UBE test date (October 5–6, 2020; February 23–‍24, 2021; or July 27–28, 2021).

In addition, for the UBEs conducted on October 5–6, 2020 and February 23–24, 2021, we selected a random, nonstatistical sample of 60 applicants from a population of 2,504 and tested whether the ExamSoft³ data file for each applicant was deleted after the UBE test, as requested by MBBE’s director, by attempting to access the applicant’s data using their case number. If there was no case number in the system, we concluded that the applicant’s data was deleted.

To determine whether all MBBE employees and board members completed the conflict of interest training during the audit period, we requested conflict of interest training certificates of completion for all MBBE employees and board members during the audit period.

Data Reliability Assessment

To assess the reliability of the data within CMS, we exchanged emails with SJC information technology employees who were responsible for the oversight of CMS to understand the data fields related to individuals who took the UBE. We also verified that the MBBE website’s references to the UBE clearly stated when the examinations were going to occur. We verified that there were no duplicate case numbers and that all entry dates were formatted as dates within our audit period in instances where the final score field in CMS was a numeric value greater than zero.

We also requested, reviewed, and verified a judgmental sample of 47 original applications received from SJC to determine whether the name on the application matched the name associated with the case number in CMS. We also verified that the application date agreed with the corresponding entry date field in CMS. 

Based on the results of these data reliability procedures, we determined the CMS and MBBE data to be sufficiently reliable for purposes of our audit.

3.    ExamSoft is a third-party subcontractor of MBBE that stores unique data on each applicant in a controlled environment with very limited access. After UBE results are finalized, ExamSoft deletes the data file for each applicant upon a request from MBBE.

Date published:	April 4, 2023