Audit of the Municipal Police Training Committee Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Municipal Police Training Committee.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Municipal Police Training Committee (MPTC) for the period January 1, 2021 through December 31, 2022.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Did MPTC develop and implement the trainings required by Chapter 69 of the Acts of 2018 (also known as the Criminal Justice Reform Act) and Chapter 253 of the Acts of 2020 (also known as the Police Reform Law), which are codified in Sections 116A–D and G–K of Chapter 6 of the General Laws?	Yes
Did MPTC ensure that all MPTC-operated and MPTC-authorized training academies delivered a standardized training curriculum as required by Section 4(f)(1) of Chapter 6E of the General Laws?	No; see Findings 1 and 2
Did MPTC process permanent exemptions to, and temporary waivers of, training requirements in accordance with Section 3.03 of Title 550 of the Code of Massachusetts Regulations (CMR)?	Yes

To accomplish our audit objectives, we gained an understanding of MPTC’s internal control environment relevant to our objectives by reviewing applicable policies and procedures and by interviewing MPTC officials. We evaluated the design and tested the operating effectiveness of internal controls related to the delivery of standardized training by MPTC-operated and MPTC-authorized training academies and the approval of exemptions and temporary waivers of training requirements. In performing our work, we found that MPTC did not have an updated internal control plan (ICP). See Finding 3.

To obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Required Training Content

To determine whether MPTC developed and implemented the trainings required by statute, we traced Sections 116A–D and G–K of Chapter 6 of the General Laws to MPTC lesson plans. Specifically, we confirmed that the lesson plans included training on the following topics:

• domestic and sexual violence complaints;
• hate crimes;
• how to use technology to increase public safety;
• the protection, care, and custody of minors upon arrest of their parents or guardians;
• bias-free policing and de-escalation and disengagement tactics, specifically for incidents impacted by people with mental health issues or developmental disabilities;
• in-service training program for school resource officers;
• training program for appropriate interactions with people on the autism spectrum or with other intellectual and developmental disabilities;
• regulation of physical force; and
• police officer mental wellness and suicide prevention.

We also reviewed emails and meeting agendas to determine whether MPTC collaborated with state agencies and other appropriate groups to develop and deliver content when required by statute.

We noted no exceptions in our testing. Therefore, we concluded that, during the audit period, MPTC developed and implemented the trainings required by Chapter 69 of the Acts of 2018 and Chapter 253 of the Acts of 2020, which are codified in Sections 116A-D and G-K of Chapter 6 of the General Laws.

Delivery of a Standardized Training Curriculum

Recruit Officer Course

We examined the Basic Recruit Curriculum Training Reports (RCTR) for all 46 Recruit Officer Course (ROC) training academies that commenced during the audit period. We did this to determine whether MPTC-operated and MPTC-authorized academies delivered MPTC-approved ROC lesson plans (each composed of 46 lessons across 800 hours) to all 1,618 student officers enrolled during the audit period. We inspected the RCTR to determine whether every lesson was delivered with the proper number of classroom and skill development hours as specified in the MPTC Academy Director’s Manual. We also inspected each lesson plan’s version number reported on the RCTR to determine whether each lesson plan was current at the time of instruction. Further, we determined whether the academy directors certified, in writing, that all student officers named on the final ROC academy roster successfully completed the prescribed course of study as described in the MPTC Annual Course Catalog.

To determine whether MPTC instructors had valid MPTC certifications for the subject matter they taught at the time the class was delivered, in accordance MPTC’s Academy Director’s Manual (Version: F2021), we judgmentally⁵ selected for testing a nonstatistical sample of 10 out of the 46 training academies that commenced during the audit period. We compared each instructor’s name listed on the RCTR to the instructor database within Acadis to determine whether the instructor was certified in the subject matter on the date that they delivered the course.

In-Service Training for Veteran Officers

We examined the Training Year In-Service Compliance Report for July 1, 2021 through June 30, 2022 generated by Acadis, which gave us a population of 17,696 officers from 452 law enforcement departments. We separated the population into two groups: 16,726 officers reported as having fulfilled the in-service training requirement (fulfilled group) and 970 officers reported as not having fulfilled their in-service training requirement by September 30, 2022⁶ (unfulfilled group). We statistically⁷ sampled the fulfilled group at a 90% confidence level⁸ with an expected error rate⁹ of 0% and a tolerable error rate¹⁰ of 5% for a sample size of 47 officers. Using the Acadis software, we inspected each fulfilled officer’s training record to confirm that the officer completed the nine-lesson curriculum of the in-service training program by June 30, 2022.

We took a random, nonstatistical sample of 60 officers from the unfulfilled group. We analyzed each unfulfilled officer’s training record to determine whether the unfulfilled status was accurate and whether any unfulfilled records were the result of the officer’s noncompliance with the annual in-service training requirement.

We determined that MPTC did not ensure that all training academies delivered a standardized ROC curriculum (see Finding 1). In addition, we found that MPTC did not ensure that all lessons within the ROC curriculum that were delivered during the audit period were taught by certified instructors (see Finding 2).

Permanent Exemptions

To determine whether MPTC processed permanent exemptions to statutory police officer training requirements in accordance with 550 CMR 3.03, we took the actions described below.

MPTC provided us with an Excel list of all 124 petitions for exemptions received by MPTC during the audit period, which listed the user identification number, name, police department, date of the board vote, examination date, and approval/denial of the exemption. We organized these petitions into three separate groups. The first group comprised petitions for exemption that were unconditionally approved by MPTC (total of 85). The second group comprised petitions for exemption that were conditionally approved (total of 16). The third group comprised petitions for exemption that were not approved (total of 23). Of the 23 petitions for exemption that were not approved, 10 were denied and 13 were classified as “not applicable,” “tabled,” or “withdrawn,” and were not presented to MPTC for consideration.

For the petitions for exemption that were unconditionally approved by MPTC, we selected a random, nonstatistical sample of 15 from the population of 85. For the petitions for exemption that were conditionally approved, we selected all 16. For the petitions for exemption that were not approved by MPTC, we selected all 10 that were denied. Combined, our total sample comprised 41 petitions for exemption.

For each of the petitions for exemption in our sample of 41, we verified that all required documents were present in the application file and that the content met the criteria stated in the regulations. We then examined MPTC monthly meeting minutes to determine whether the exemption applications were presented to and approved by both the MPTC Standards Subcommittee and the full MPTC.

We noted no exceptions in our testing. Therefore, we concluded that, during the audit period, MPTC processed permanent exemptions to training requirements in accordance with 550 CMR 3.03.

Temporary Waivers

To determine whether MPTC processed waivers of statutory police officer training requirements in accordance with 550 CMR 3.03, we took the actions described below.

MPTC provided us with an Excel list of all 71 petitions for temporary waivers presented to MPTC during the audit period, which listed the user identification number, name, academy, date of the board vote, examination date, and granting/denial of the waiver. We organized these petitions into two separate groups: 66 waivers that were granted by MPTC and 5 that were denied. We examined each application file to determine whether the required documents were present and that the content met the criteria stated in the regulations. We then examined the MPTC monthly meeting minutes to determine whether the waiver applications were presented to and approved by both the MPTC Standards Subcommittee and full MPTC.

We noted no exceptions in our testing. Therefore, we concluded that, during the audit period, MPTC processed waivers of training requirements in accordance with 550 CMR 3.03.

We used a combination of statistical and nonstatistical sampling methods for testing, and we did not project the results of our testing to any corresponding populations.

Data Reliability Assessment

Acadis Readiness Suite

To determine the reliability of data obtained from the Acadis database, we reviewed the Federal Risk and Authorization Management Program Annual Security Assessment Reports¹¹ performed by Kratos that were published on April 9, 2021 and April 9, 2022 and ensured that certain information system control tests had been performed. In addition, we tested certain general information technology controls (security management, access control, configuration management, segregation of duties, and contingency planning) for the period January 1, 2021 through December 31, 2022. In addition, we tested the application control over attendance to ensure that the officer viewed the slides presented in online training to receive credit for watching the required video lessons.¹²

New Recruits

To determine the reliability of the new recruit data from the Acadis database, we took the following actions. We extracted from the database a list of student officers who were enrolled in ROC academies during the audit period and grouped these student officers by the ROC academy in which they were enrolled. We then compared the names of those student officers to the enrollment rosters contained on the ROC After Action Reports stored in SharePoint for each of the 46 ROC training academies that commenced during the audit period. In addition, we traced 10 training academies from the After Action Reports to the Acadis database and traced 10 training academies from the Acadis database to the After Action Reports to confirm that each academy name and start date and end date of the academy was consistent.

Veteran Officers

To determine the reliability of the list of veteran officers in the Acadis database, we compared the Acadis list of veteran officers at 452 police departments to the list of required in-service veteran officers produced by the Massachusetts Peace Officer Standards and Training Commission.

Mptctraining.com

Mptctraining.com is a repository software that controls the distribution of the recruit training standardized lesson plans. We assessed the reliability of the data obtained from mptctraining.com by testing certain general information technology controls (security management, access control, configuration management, and segregation of duties) for the period January 1, 2021 through December 31, 2022.

Waivers and Exemptions

To determine the reliability of the lists of petitions for temporary waivers and exemptions presented to MPTC during the audit period, which MPTC provided to us, we checked both lists for hidden rows and columns. To test the accuracy of the lists, we selected a random sample of 20 (10 from each list) police officers from the lists and compared these officers’ names, sponsoring agencies, and dates of petitions to source documentation (hardcopy petitions) provided by MPTC. To test the completeness of the lists, we judgmentally selected a different set of 20 (10 from each list) police officers from source documentation at the MPTC office (hardcopy petitions) and traced these back to the lists.

Based on the results of the data reliability assessment procedures described above, we determined that the information we obtained was sufficiently reliable for the purposes of our audit.

5.    Auditors use judgmental sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review. Auditors use their knowledge and judgment to select the most appropriate sample. For example, an auditor might select items from areas of high risk. The results of testing using judgmental sampling cannot be used to make conclusions or projections about entire populations; however, they can be used to identify specific issues, risks, or weaknesses.

6.    Although the in-service training year ends June 30, MPTC allows a 90-day reporting grace period for in-service training delivered at MPTC-authorized training locations.

7.    Auditors use statistical sampling to select items for audit testing when a population is large and contains similar items. Auditors generally use a statistical software program to choose a random sample when sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

8.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

9.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

10.    The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

11.    A Federal Risk and Authorization Management Program Annual Security Assessment Report is an annual security assessment of certain software to make risk-based decisions. Use of this assessment is required by the Federal Information Security Management Act of 2002.

12.    Acadis has built-in system controls designed to prevent officers from skipping ahead or bypassing certain sections of the online training content. We tested these controls and determined that they were in place and working as intended during the audit period. However, on November 15, 2024, MPTC notified all chiefs of police departments that it identified instances where officers were using outside technology to override these controls. Certain fiscal year 2025 trainings (outside of the audit period) that should have taken hours to complete were being completed in a matter of minutes. MPTC officials told us that this was still an ongoing investigation.

Date published:	December 26, 2024