Overview
During the audit period, NWDA did not consistently collect and analyze data on its Juvenile Diversion Program (JDP). Further, NWDA did not provide evidence that it performed an analysis of the data it collected, or that action was taken to improve the outcome of the program. These are all recommended by the “Juvenile Diversion Guidebook: Prepared by the Models for Change Juvenile Diversion Workgroup,” the “Massachusetts Youth Diversion Program: Model Program Guide,” and our 2018 audit report of NWDA. Additionally, our review of juvenile diversion case files found that NWDA did not always maintain key documentation regarding juveniles’ participation in the program.
In our prior audit of NWDA (No. 2018-1260-3J, dated November 29, 2018), the audit team found that the JDP “does not collect data or conduct post-completion outcome evaluations to determine the program’s effectiveness.” During our post-audit review, to verify that the recommendations made in the 2018 audit were being implemented, NWDA stated that it would (1) continue to collect statistical data and use the post-completion surveys of parents/guardians and juveniles in order to improve the effectiveness of the JDP and (2) would develop plans to expand the tracking and analysis of data to improve the outcomes of juveniles in the program.
During our audit, NWDA provided us with data, charts, and statistics for the audit period covering the following topics:
- the number and percentages of juveniles who completed and did not complete the JDP (2021, 2022, and 2023);
- the demographics of juveniles in the program, broken down by gender, race, and ethnicity (2021);
- the average time juveniles took to complete or fail a diversion program (2021, 2022, and 2023); and
- the number of juveniles by court who participated in the JDP, the number of juveniles who reoffended within one year of participating in the program, and the percentage of juvenile recidivism by year (2021, 2022, and 2023).
The data was generated mostly from data extracted from the District Attorney Management Information Office Network (DAMION). We also received a data assessment performed by NWDA to determine whether juveniles reoffended within one year of their juvenile diversion completion date.
Our review of the provided documentation revealed the following:
- NWDA did not collect or analyze all diversion data on a yearly basis.
- Data collected and analyzed was prepared during the year 2024, after the start of our audit, when this audit topic had been communicated to NWDA, and was not consistently collected and analyzed in the years in which the diversion happened.
Additionally, although NWDA did provide us evidence that it assessed this data to take action to improve the outcome of the program, this assessment may not be effective if it was based on partial data collected only.
The results of our review of the sample of 35 JDP case files identified the following:
- out of the 35 cases, 2 cases (6%) did not have a waiver of counsel;
- out of the 35 cases, 4 cases (11%) did not have evidence of monitoring of juveniles’ progress in the program; and
- out of the 35 cases, 1 case (3%) did not have a completion letter.
By not collecting sufficient JDP data, NWDA limits its ability to achieve its goal of improving program outcomes and reducing recidivism. Without generating and retaining adequate documentation, NWDA cannot ensure that it follows best practices and procedures for an effective diversion program.
Authoritative Guidance
The Models for Change Juvenile Diversion Workgroup’s Juvenile Diversion Guidebook recommends collecting data for outcome evaluation as part of a logic model for diversion programs. This data includes the number of juveniles referred and diverted, the number of juveniles successfully completing terms of diversion, the hours of community services assigned and completed, and the number of diverted juveniles who commit new offenses while on diversion status.
The Juvenile Justice Policy and Data Board’s10 Massachusetts Youth Diversion Program: Model Program Guide, in March 2021, also highlights the importance of collecting and analyzing data that help staff members be aware of any issues in the program and facilitate course correction. This guide also details the types of inputs, outputs, operational measures, and outcomes that could be collected and analyzed.
Regarding the administration of the program, the “Northwestern District Attorney’s Juvenile Diversion Program Policies and Procedures” states the following:
The JDP Consent, Authorization, Waiver, and Acknowledgement Form must be signed by the juvenile and parent/guardian prior to the intake. The waiver shall be explained to the juvenile and parent/guardian by an attorney appointed by the court. . . .
The JDP Specialist will monitor through the counselor to check the progress and confirm whether other services are recommended or if there are other concerns. The JDP Specialist will discuss with the counselor at the beginning of the case the plan for communication. . . .
Community service should also be monitored. JDP Specialist will work to be creative with plan for Community Service. . . .
[The Juvenile Specialist will] contact each family to notify them of successful completion and dismissal of case. Send a congratulatory letter within a 30-day period after program completion.
Reasons for Issue
Regarding the collection of data and statistics for program improvement, NWDA stated that DAMION does not allow NWDA to track all categories of information that NWDA would ideally like to track. NWDA also told us that the years impacted by COVID-19 impacted the trends and outcomes of the program.
Regarding missing documentation of monitoring juvenile compliance with diversion requirements, NWDA said that some monitoring is performed with juveniles and their parents through text messages and telephone calls, which can be difficult to document. Additionally, sometimes juveniles’ attorneys do not provide documentation, such as the waiver of counsel, to NWDA. However, NWDA could use a form to log text messages and telephone calls and the topics discussed with juveniles and their parents. Also, it is important for NWDA to collect waivers of counsel from juveniles’ attorneys, even if the attorneys do not send these documents to NWDA. NWDA should therefore make additional efforts to secure these documents and should document and retain records of these efforts.
Recommendations
- NWDA should consistently collect JDP data and analyze it on an annual basis and use the results for program improvement. NWDA should identify alternatives for data collection if the District Attorney Management Information Office Network (DAMION) does not suit NWDA’s data collection.
- NWDA should create and implement adequate controls to ensure that it collects, records, and retains all supporting documentation related to JDP activities.
Auditee’s Response
We appreciate the Auditor’s recognition that the NWDA collects and analyzes diversion program data and assesses this data to take action to improve the Juvenile Diversion Program (JDP), which is a discretionary program run by the NWDA to divert eligible youth from the criminal justice system.
The central goal of the JDP as stated in its Policies and Procedures Manual is:
To offer intervention at an early stage to eligible juvenile offenders in order to increase safety, accountability, and competency development; improve outcomes; and decrease the likelihood of further offenses.
We believe that we meet the objectives contained in this goal by effectively measuring outcomes for juveniles who participate in the program and by maintaining data on these outcomes. Our central criteria for measuring a successful outcome is focused on the development and completion of an individualized diversion plan that is restorative for victims, juveniles, their families, and their communities.
The JDP collects data and assesses its effectiveness in meeting its core goals and objectives in many ways, including: comprehensive assessments of a juvenile and development of an individualized diversion plan that is monitored and the outcomes are tracked; regular assessment and review of the program within the organizational structure of our office; staff participation in professional development in best practices; collaborative work with community partners; and through data collection within internal office databases.
Since our last audit in 2018, the NWDA has made advances in data collection and analysis for the Juvenile Diversion Program. We have prepared report from the available data and used this data to improve the program. We developed a working group and met to develop an appropriate plan for collecting and analyzing our data. Meanwhile, the juvenile justice system, and specifically juvenile diversion, underwent significant reform by way of legislative and judicial actions. These changes and the resulting guidance offered further support to our opinion that developing appropriate metrics to track our juvenile diversion program with a goal of identifying challenges/successes and measuring outcomes is an incredibly complex and challenging endeavor. This is in large part due to the short nature of the diversion program and our limited pool of participants each year given our small rural jurisdiction. We are encouraged by our recidivism statistics that note a low percentage of re-offense after successful completion of the program, but we continue to seek more opportunities for data analysis that will support us in our work. In 2024, we engaged the services of a Data Analyst to assist us in collecting and analyzing data about our JDP from DAMION for each of the past five years. We are currently in the process of analyzing the data provided and determining what changes to our program are warranted. We also continue to explore alternative data collection systems to DAMION to allow for a more expansive collection and review in the future. We remain committed to building on the success of our well-established program with additional data expansion.
We also appreciate the Auditor’s recognition that the NWDA had all expected and appropriate documentation in most of the files reviewed. We recognize the need to ensure every file is appropriately documented and are currently using a case monitoring notebook system to ensure all activities related to compliance with the diversion program are able to be recorded contemporaneously by the Diversion Specialist and included in this single locations for future cases.
Auditor’s Reply
We encourage NWDA to continue to take the necessary steps to consistently collect JDP data and analyze it on an annual basis and use the results for program improvement. Only through the consistent collection and analysis of data can NWDA assess the effectiveness of satisfying its stated goals. Based on its response, the NWDA is taking measures to address our concerns regarding the collection, recording, and retention of all supporting documentation related to Juvenile Diversion Program activities.
| Date published: | December 30, 2024 |
|---|