Audit of the Executive Office of Public Safety and Security Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Executive Office of Public Safety and Security.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Executive Office of Public Safety and Security (EOPSS) for the period July 1, 2020 through October 31, 2022.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
1. Did EOPSS ensure that the information entered into the Provider Sexual Crime Report (PSCR) master database did not contain personally identifiable information (PII) of survivors of sexual assault, in accordance with Section 12A1/2 of Chapter 112 of the General Laws?	No; see Finding 1
2. Did EOPSS ensure that all previously untested investigatory sexual assault evidence collection kits (SAECKs) were reviewed for quantity-limited evidence (QLIM) and that district attorneys’ (DAs’) offices were notified of the review results within 90 days of the effective date of Section 2(a) of Chapter 35 of the Acts of 2021?	No; see Finding 2
3. Did EOPSS ensure that the Massachusetts State Police Crime Laboratory (MSPCL) shipped to an accredited private crime laboratory all previously untested investigatory SAECKs that were identified as not containing QLIM within 180 days of the effective date of Section 2(b) of Chapter 35 of the Acts of 2021?	No; see Finding 3
4. Did EOPSS file “Sexual Assault Evidence Collection Kit (SAECK) Quarterly Reports” with the Legislature, as required by Section 2(c) of Chapter 35 of the Acts of 2021?	Yes
5. Did EOPSS ensure that the location and status of SAECKs were accurate in the Track-Kit system, in accordance with Section 18X(b)(i) of Chapter 6A of the General Laws?	No; see Finding 4

To accomplish our audit objectives, we gained an understanding of the aspects of EOPSS’s internal control environment that we determined to be relevant to our objectives by reviewing applicable agency policies and procedures, as well as by interviewing EOPSS staff members and management. We evaluated the design of controls over the PSCR master database, previously untested investigatory SAECKs, EOPSS’s “Sexual Assault Evidence Collection Kit (SAECK) Quarterly Reports,” and the Track-Kit system. To obtain sufficient, appropriate evidence to address our audit objectives, we performed the following procedures.

PSCR Data Review for PII

To determine whether the PSCR master database contained any PII, we obtained a list from the Office of Grants and Research (OGR) of all the records in the PSCR database. There was a total of 27,028 records in this list from the period August 15, 1999 through March 29, 2023.

We performed a data analysis on all of these records to determine whether they included either a name or address, producing a population of 2,066 records that could potentially contain PII. We determined that 322 of these records included a street number and street name that could be combined with the city and state listed to complete a physical address. We filtered this population of 322 records for records that had a SAECK number within the data, producing 236 records with address information and an associated case file. From the population of 236 records, we selected a random, nonstatistical⁴ sample of 35 records. For each of the 35 records in this sample, we reviewed the physical sexual assault case files from MSPCL, including police reports and law enforcement correspondence, to determine whether the address in the data matched that of the survivor.

In addition, of the 2,066 records that could potentially contain PII, we determined that 14 records included both a first and last name. However, the names were not identified as being law enforcement officers, medical providers, or employees of the Department of Social Services, which would be an acceptable release of information. Further, we reviewed the associated sexual assault case file for all 14 records to determine whether the information matched that of the survivor or a relative of the survivor.

Due to the sensitive nature of the aforementioned information, we took measures to keep this information confidential. Those measures include, but are not limited to, redacting any PII contained within records we collected and only collecting those records that were necessary to serve as audit evidence.

See Finding 1 for an issue we identified with PII in the PSCR master database.

QLIM Review and Notification

To determine the timeliness of the QLIM review, EOPSS provided us with a Microsoft Excel workbook that logged the progress of the population of 6,502 previously untested investigatory SAECKs maintained by MSPCL. MSPCL told us that the workbook contained two “Review Date” data fields that signified the dates MSPCL completed its initial and secondary reviews of each SAECK for QLIM. The Acts of 2021, which went into effect July 1, 2021, mandated that the SAECKs be reviewed within 90 days; therefore, we determined that the deadline was September 29, 2021. We calculated the number of days between each SAECK’s most recent review date and September 29, 2021 to determine whether each review was completed on time.

To determine the timeliness of each MSPCL QLIM review notification to the assigned DA’s office, we extracted a population of 2,090 SAECKs containing QLIM from the original population of 6,502 previously untested investigatory SAECKs. We reviewed all 75 notification emails and attachments sent from MSPCL to the assigned DAs’ offices informing them of the results of its review. We confirmed that each of the 2,090 SAECKs containing QLIM was included in these emails. We compared the email dates to the deadline date to determine whether the emails were sent within the 90-day timeframe.

See Finding 2 for an issue we identified with the QLIM reviews within the 90-day timeframe.

Transfer of SAECKs within 180 Days

We determined the timeliness of MSPCL’s transfer of previously untested investigatory SAECKs that were identified as not containing QLIM to an accredited private crime laboratory for DNA testing. EOPSS provided us with a Microsoft Excel workbook that recorded the progress of the 6,502 previously untested investigatory SAECKs. Our analysis of the “Testing Status” data field identified 2,819 records with values that included shipping dates (month and year) to the accredited private crime laboratory from January 2022 through December 2022. Of the corresponding 2,819 SAECKs, we found 1,834 of these SAECKs that were identified as not containing QLIM. These served as our population of shipped previously untested investigatory SAECKs that were identified as not containing QLIM. See the Appendix for a breakdown of the 6,502 SAECKs that were previously collected but had not yet received DNA testing.

The data field “Date of DA Response” signified the date an assigned DA’s office notified the crime laboratory of its approval for a kit to proceed with DNA testing. We calculated the deadline of 180 days from the effective date of the Acts of 2021 to be December 28, 2021. We selected a sample of 527 previously untested investigatory SAECKs from the population of 1,834 that had DA approval for DNA testing as of that date to determine whether these SAECKs were shipped to an accredited private crime laboratory by the deadline.

We also interviewed knowledgeable staff members at MSPCL to determine when the first shipments of previously untested investigatory SAECKs were sent to a private crime laboratory. Additionally, we verified the accreditation certificates for both MSPCL and the private crime laboratory were valid during the audit period.

See Finding 3 for an issue we identified with the transfer of SAECKs within the 180-day timeframe.

Submission of Quarterly Reports

To determine whether EOPSS filed “Sexual Assault Evidence Collection Kit (SAECK) Quarterly Reports” as required, we obtained copies of all five reports that reported on activity related to previously untested investigatory SAECKs that occurred during the audit period, as well as the corresponding emails from EOPSS to the House of Representatives, the Senate, and the Joint Committee on Public Safety and Homeland Security. In addition, we examined the Journals of the House of Representatives and the Senate, which document the resolutions, orders, petitions, and reports submitted to the House and the Senate, to verify that EOPSS submitted the “Sexual Assault Evidence Collection Kit (SAECK) Quarterly Reports.” We examined all five quarterly reports to determine whether they included the following information required by Section 2 of Chapter 35 of the Acts of 2021:

(i) the number of untested investigatory sexual assault evidence kits in the possession of public crime laboratories prior to [July 1, 2021]; (ii) the year each kit was collected; (iii) the date each kit was tested; and (iv) the date the resulting information was entered into [the Combined DNA Index System (CODIS)] and the state DNA databases.

We noted no exceptions in our testing; therefore, we concluded that, during the audit period, EOPSS filed “Sexual Assault Evidence Collection Kit (SAECK) Quarterly Reports” with the Legislature, as required by Section 2(c) of Chapter 35 of the Acts of 2021.

Track-Kit System

To determine whether the Track-Kit system correctly tracked the location and status of SAECKs throughout the criminal justice process in accordance with Section 18X(b)(i) of Chapter 6A of the General Laws, we obtained a list of all 3,547 records⁵ from the Track-Kit system. Within this population, we identified 3,008 SAECKs with collection dates entered during the audit period. Based on our internal control evaluation, we selected a random, statistical⁶ sample of 60 records using a 95% confidence level,⁷ 5% tolerable rate,⁸ and 0% expected error rate.⁹ For each of the 60 records in our sample, we visited the associated crime laboratory or local law enforcement agency (LLEA) recorded as the current location of the SAECK and confirmed with officials that the kit was present. SAECKs held in a location other than what was on file were considered instances of noncompliance.

See Finding 4 for an issue we identified with location updating within the Track-Kit system.

We used a combination of nonstatistical and statistical sampling methods for our audit objectives and did not project the results from the samples to the populations.

Data Reliability Assessment

PSCR Master Database

To determine the reliability of the list of records from the PSCR master database, we performed the following tests:

• We conducted a process walkthrough, observing OGR analysts entering information from a PSCR form into the PSCR master database.
• We checked for duplicate records and missing values in key fields.
• We compared the total number of PSCR records on the list we received from OGR against the total number of records we observed in the PSCR master database to ensure that we were provided the complete list.
• We randomly selected a sample of 20 records from the list of records and traced the information to source documentation (copies of the original PSCR forms that OGR received from medical facilities) to ensure accuracy.
• We randomly selected a sample of 20 PSCR forms and traced information from the forms to the database to ensure completeness.

Previously Untested Investigatory SAECKs and Quarterly Reports

To determine the reliability of the list of previously untested investigatory SAECKs that we obtained from MSPCL, we performed the following tests:

• We interviewed EOPSS and MSPCL management who were responsible for the data in this list.
• We checked for duplicate records.
• We compared the total number of records in this list to the totals reported by each DA’s office.
• We compared the total number of records in this list to the totals reported in EOPSS’s February 2023 “Sexual Assault Evidence Collection Kit (SAECK) Quarterly Report.”
• We randomly selected a sample of 20 records from the list of previously untested investigatory SAECKs and traced the information (e.g., case number, SAECK barcode number, law enforcement agency, and SAECK shipping date) to source documentation (e.g., Evidence Submission forms, Requests for the Examination of Physical Evidence forms, and Record of Evidence Submitted forms) included in physical MSPCL case files to ensure accuracy.
• We randomly selected a sample of 20 records from MSPCL case files and traced the information to the list of previously untested investigatory SAECKs we obtained from MSPCL to ensure completeness.

Track-Kit System

To determine the reliability of the Track-Kit system data, we interviewed EOPSS and MSPCL management who were responsible for maintaining the data. We also reviewed the System and Organization Control reports¹⁰ that covered the period January 1, 2021 through December 31, 2021. We verified that the System and Organization Control reports described testing of certain information system general controls (access controls, security management, configuration management, contingency planning, and segregation of duties) without exception. In addition, we reviewed the peer review report of the accounting firm that prepared the Service Organization Control reports.

In addition, we performed the following tests:

• We checked for duplicate records, missing values in key fields, and dates outside the audit period.
• We compared the total number of records in the list we received from EOPSS to the agency’s total number of records reported in the system to ensure that we were provided a complete list.
• We randomly selected a sample of 20 records from the data and traced the information (e.g., unique SAECK bar code number, medical facility, and the LLEA) to the PSCR forms to ensure accuracy, and we traced the information from 20 randomly selected PSCR forms in physical files to the Track-Kit system data to ensure completeness.

Based on the results of the data reliability assessment procedures described above, we determined that the information obtained for the audit period was sufficiently reliable for the purposes of our audit.

4.    Auditors use nonstatistical sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review.

5.    This population did not include the previously untested investigatory SAECKs, as those kits were not in the Track-Kit System.

6.    Auditors use statistical sampling to select items for audit testing when a population is large (usually over 1,000) and contains similar items. Auditors generally use a statistics software program to choose a random sample when statistical sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

7.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

8.    The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

9.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

10.    A System and Organization Control report is a report, issued by an independent contractor, on controls about a service organization’s systems relevant to security, availability, processing integrity, confidentiality, or privacy.

Date published:	August 8, 2024