Overview
Commonwealth Corporation (CommCorp) did not consistently collect or analyze employment outcome data related to its YouthWorks program participants. Specifically, during the audit period, YouthWorks grant recipients had the ability to report in the YouthWorks database whether each corresponding program participant had obtained unsubsidized employment or been retained by a YouthWorks partner. However, we found that CommCorp does not require that YouthWorks grant recipients report this employment outcome data. We found that, of the 10,280 YouthWorks participant spots filled during the audit period, only 179 (2%) had data recorded in the YouthWorks database indicating that each corresponding program participant had obtained unsubsidized employment or been retained by a YouthWorks partner.
If CommCorp does not track its program participants’ employment outcome data, then it cannot evaluate and understand the impact of YouthWorks on the individuals who use the program. This prevents the public and policymakers from determining the effectiveness of this program and use of taxpayer dollars. It also prevents YouthWorks from demonstrating its value to secure additional funding.
Authoritative Guidance
Section 116(b)(2)(A) of the Workforce Innovation and Opportunity Act (WIOA) of 2014 established three employment-related accountability indicators to assess the effectiveness of youth workforce development programs. WIOA identifies the following indicators of program performance:
(I) the percentage of program participants who are in unsubsidized employment during the second quarter after exit from the program;
(II) the percentage of program participants who are in unsubsidized employment during the fourth quarter after exit from the program;
(III) the median earnings of program participants who are in unsubsidized employment during the second quarter after exit from the program.
Although CommCorp is not required to use these indicators, we consider it a best practice.
The US Department of Labor provides guidance to state workforce development agencies on how to implement the performance accountability requirements under WIOA through a document called “Training and Employment Guidance Letter WIOA No. 26-16.” This document states,
Matching a participant’s [Social Security number] against quarterly wage record information is the most effective means for determining employment status and earnings for a program participant.
Although CommCorp is not required to follow this guidance, we consider it a best practice.
Section VIII (for fiscal year 2021) and Section VII (for fiscal year 2022) of CommCorp’s YouthWorks “Program Administration and Management Guide” states,
Commonwealth Corporation will analyze and share wage record matching data that looks at the earnings of YouthWorks participants for eight quarters after their participation in the program in comparison to wages by similar demographic of non-participants.
During the audit period, CommCorp had access to wage records through a memorandum of understanding with the Department of Unemployment Assistance. This memorandum of understanding states,
The purpose [of this agreement] is to use matched wage reporting data to track the post-program employment outcomes of participants in YouthWorks . . . programs that are funded through state appropriations and administered by Commonwealth Corporation. Analyses will be conducted for each program at the statewide level as well as by grantee and Workforce Board region.
However, during the audit period, CommCorp did not use this memorandum to obtain available wage records, which prevented it from demonstrating the cost-benefit of YouthWorks.
Reasons for Issue
CommCorp states in its YouthWorks “Program Administration and Management Guide” that it “will analyze and share wage record matching data that looks at the earnings of YouthWorks participants for eight quarters after their participation in the program.” However, it does not have documented procedures on how it will do so. Also, even though the policy exists, CommCorp did not appear to follow it during the audit period. CommCorp officials told us that before the audit period, CommCorp did conduct wage tracking to determine whether YouthWorks program participants secured unsubsidized employment after the program. However, this process was paused during the audit period because of insufficient staffing resources.
During the audit period, CommCorp had access to wage records11 through a memorandum of understanding with the Massachusetts Department of Unemployment Assistance. However, CommCorp officials told us that they did not use these records to evaluate the employment status or earnings of former YouthWorks program participants because the COVID-19 pandemic made it difficult for CommCorp to obtain data it needed from the Department of Unemployment Assistance to perform this analysis.
CommCorp officials told us that they use post-program surveys of YouthWorks program participants as a method for measuring the success of its YouthWorks program. However, we found that these post-program surveys only allow for a self-assessment of the skills that program participants gained because of YouthWorks and do not capture whether a program participant gained unsubsidized employment as a result of YouthWorks.
Recommendations
- CommCorp should develop policies and procedures to effectively monitor the extent to which its YouthWorks program achieves its intended purpose of helping program participants secure unsubsidized employment. These policies and procedures should include information on how to routinely collect and analyze employment outcome data related to its YouthWorks program participants.
- CommCorp should revise its YouthWorks post-program survey to capture information regarding whether a program participant gained unsubsidized employment as a result of its YouthWorks program.
- CommCorp should require all YouthWorks grant recipients to report employment outcome data in the YouthWorks database.
Auditee’s Response
CommCorp’s YouthWorks program experienced tremendous changes during the audit period, which coincided with the Governor’s Declared State of Emergency in response to the COVID-19 pandemic. These changes included a complete program overhaul, transitioning from an in-person summer jobs program pre-pandemic, to a fully remote year-round program. In addition to the program re-design, there was staff turnover on the YouthWorks team during the audit period that presented further challenges with regard to consistent enforcement of established program protocols, policies, and procedures.
YouthWorks is in the process of re-tooling its review strategy to analyze participant post-program outcomes. YouthWorks has an active Alumni Engagement network that allows the CommCorp YouthWorks team to obtain post-program participant data on a voluntary basis. CommCorp is also in the process of both enhancing its program design and augmenting related review metrics to ensure that these changes are fully implemented by the start of the next program cycle (contracts beginning on or after May 15, 2024). The YouthWorks Program revised its post-program survey, which now includes participant employment and education outcomes, and implemented the new survey during the last program cycle (October 2023). . . .
YouthWorks is in the final stages of developing a new secure database, which will be implemented on June 10, 2024. That database will have robust internal controls and safeguards to ensure that all required participant data is collected, and any incomplete entries into the database will serve as a barrier to complete program enrollment. The new database will also ensure more complete and accurate participant outcome data, as a participant profile will not be complete until all wage related data is entered. Such data will be verified against the Department of Unemployment Assistance’s recorded wage data. A revised [memorandum of understanding] between CommCorp and the [Department of Unemployment Assistance] will support this work. Grant recipients will be able to enter employment outcome data in the new database.
Auditor’s Reply
Based on its response, CommCorp is taking measures to address our concerns on this matter.
| Date published: | July 9, 2024 |
|---|