Audit of the Department of Youth Services Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Department of Youth Services

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Department of Youth Services (DYS) for the period July 1, 2017 through June 30, 2019.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Did caseworkers meet with each of their assigned youths with a frequency and method that are in accordance with applicable policies in the DYS Case Management Practice and Procedure Manual?	Yes
Did DYS assign a maximum of 15 caseloads to each of its caseworkers?	No; see Finding 1
Did DYS administer a gang training program for its newly hired employees in accordance with the United States Department of Justice Office of Juvenile Justice and Delinquency Prevention’s “Best Practices to Address Community Gang Problems”?	Yes
Did DYS take adequate steps to prevent the reoccurrence of all supported 51B reports of abuse and/or neglect of youths in its care and custody in accordance with Section 4.47 of Title 110 of the Code of Massachusetts Regulations?	Partially; see Other Matters

 

Additionally, DYS did not always revoke terminated employees’ user access in a timely manner (Finding 2).

To achieve our audit objectives, we gained an understanding of the internal control environment we determined to be relevant to the objectives by conducting site visits and inquiries with DYS officials and by reviewing DYS’s internal control plan and applicable agency policies and procedures.

We reviewed and tested the operating effectiveness of internal controls related to the monitoring of basic training attendance for DYS’s gang training program. Specifically, we performed control testing to verify that attendance sheets were signed by all DYS employees and DYS vendor employees hired during the audit period and then reviewed by DYS’s gang training instructor.

To obtain sufficient, appropriate evidence to address our audit objectives, we conducted further audit testing as described below.

Caseworker Meetings

To determine whether DYS caseworkers met with their assigned youths in accordance with DYS policies and procedures, we selected a random, statistical sample of 60 youth-weeks³ from an estimated total population of 48,036 youth-weeks during the audit period for youths who were scored as “moderate,” “high,” or “very high” risk according to their assessed Youth Level of Service / Case Management Inventory (YLS/CMI) risk levels. We also selected a second random, statistical sample of 60 youth-months from an estimated total population of 3,030 youth-months⁴ during the audit period for youths who were scored as “low” risk according to their assessed YLS/CMI risk levels.

To determine whether caseworkers met with their assigned youths according to DYS policy, we selected two random, statistical samples as discussed above. Caseworker meeting requirements vary depending on youths’ YLS/CMI risk levels: youths assessed as “moderate,” “high,” or “very high” risk require two to three weekly contacts with their assigned caseworkers, and those assessed as “low” risk require at least one monthly contact with their assigned caseworkers. Therefore, we took one sample for each of these groups and reviewed DYS’s placement log to determine the physical location of each youth during the sampled week or month. To determine whether youths aged out⁵ of DYS, we reviewed their dates of birth and verified them using independent court documents. We then reviewed the case management notes in the Juvenile Justice Enterprise Management System (JJEMS) to determine whether caseworkers met with their assigned youths in accordance with DYS policy.

Caseworker Caseloads

To determine the size of the caseload for each DYS caseworker, we reviewed all caseloads for each of the 93 caseworkers DYS employed during the audit period. For each caseworker, we analyzed the number of youths assigned for every month of the audit period. Based on interviews with DYS officials who stated that 12 to 15 cases per caseworker was a reasonable caseload for caseworkers, we determined that each caseworker should have been assigned a maximum of 15 cases. We excluded some cases from our review when a youth was temporarily reassigned to another caseworker, e.g., a caseworker was on vacation. We also excluded youths who were assigned to a caseworker for two weeks or less.

Gang Training Program

To determine whether DYS administered a gang training program during the audit period for its employees and its vendors’ employees in accordance with best practices, we selected a random, statistical sample of 40 newly hired employees from a total population of 1,158. We then obtained the gang training materials DYS used as part of its basic training for newly hired employees and examined their certificates as evidence of their gang training attendance and completion.

Follow-up Actions for Supported Reports of Abuse and/or Neglect

To determine whether DYS took adequate steps to prevent the reoccurrence of all supported reports of abuse and/or neglect of youths in its care and custody in accordance with state regulations, we initially obtained all 51A and 51B reports from the Department of Children and Families (DCF) that involved youths in DYS care and custody during the audit period. We then obtained 134 unique⁶ 51A reports from DYS to ensure that we had received all reports that involved youths in DYS care and custody during the audit period.

From the 51A reports we received from DCF, we selected all screened-in 51A reports and all of the corresponding supported 51B reports where DCF had determined that it had reasonable cause to believe that the abuse and/or neglect occurred. We then reviewed any follow-up action plans that were listed in the 51B reports, as well as evidence of their implementation from DYS. We also requested and examined evidence of any internal investigations or additional actions by DYS that would prevent reoccurrence of abuse and/or neglect of youths in its care and custody.

Data Reliability

To gain an understanding of JJEMS and its controls, we interviewed information technology (IT) personnel who were responsible for oversight of the system. To assess the reliability of the data in the system, we tested the following general IT controls: security management, access controls, configuration management, contingency planning, and segregation of duties. Further, we tested automated and manual application controls in JJEMS that were relevant to our audit objectives. In addition, we selected a random sample of 20 committed youths in DYS custody and verified their full names, identification numbers, dates of birth, and home regions from the original source documents. We could not determine the exact number of caseworker meetings held during the audit period because some youths in DYS custody were transferred into and out of residential treatment facilities and/or houses of correction and others were released to police custody for court hearings or could have aged out of DYS; DYS was not required to meet with those who had been released. Based on the procedures performed, with the exception noted in Finding 2, we determined the JJEMS data were sufficiently reliable for the purposes of our audit.

We assessed the completeness and accuracy of the 51A reports obtained from DYS Investigation Unit personnel’s emails by tracing the 51A report identification numbers to 51A reports received from DCF. We determined that the population of 51A reports received from DYS was sufficiently reliable for the purposes of our audit.

We assessed the completeness and accuracy of the list of newly hired DYS employees by comparing it to a report from the Human Resources Compensation Management System (a tool state employees use to enter their timesheets and access other human resources functions) that listed all DYS employees during the audit period. We also tested for duplicate and missing data and obtained the reconciliation from DYS for any noted discrepancies. We determined that the population of newly hired DYS employees received from DYS was sufficiently reliable for the purposes of our audit.

3.    A youth-week is a seven-day period in which a caseworker visits with an assigned youth. The number of visits depends on the youth’s risk level.

4.    A youth-month is a 30-day period in which a caseworker must have at least one visit with an assigned youth.

5.    “Aged out” means having reached a certain age (which varies with the type of offense) and no longer being required to be in DYS’s care and custody.

6.    Multiple mandated reporters can file individual 51A reports for the same incident. DCF combines such reports to conduct one investigation into any one incident. Therefore, for the purpose of our testing, if multiple 51A reports were filed for the same incident, we counted them as one unique report.

Date published:	March 14, 2022