Overview
In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Department of Elementary and Secondary Education (DESE) for the period July 1, 2018 through June 30, 2019.
We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and where each objective is discussed in the audit findings.
|
Objective |
Conclusion |
|
|
|
To achieve our objectives, we gained an understanding of DESE’s internal control environment related to our audit objectives by reviewing applicable laws and agency policies and procedures, as well as conducting inquiries with DESE management.
PQRs and SVs
To determine whether DESE conducted AE provider PQRs and SVs in accordance with its policies and procedures, we obtained a cohort list (a list of cohort assignments maintained as an Excel spreadsheet by DESE’s Adult and Community Learning Services Unit) from DESE that included both the request for proposals (RFP) application score and the cohort assignment for each of the 77 AE providers that were active during our audit period. We compared the cohort assignments to determine whether the providers were assigned to cohorts that were consistent with their RFP application scores.
Only AE providers assigned to cohort 1 were required to receive PQRs in fiscal year 2019. We inspected all 25 PQR reports for fiscal year 2019 to determine whether DESE completed the four key components of the PQR process—specifically, that all providers with PQRs were in cohort 1, all PQRs were reviewed by a supervisor, the predefined 4 of 10 indicators of program quality were assessed by the PQR team, and all reports were sent to the providers with recommendations.
Finally, we obtained a list of all 52 providers that were required to receive SVs during fiscal year 2019 (cohorts 2 and 3). From those, we selected a nonstatistical, random sample of 20 providers. We examined SV reports to assess whether these SVs were completed and the reports were reviewed by supervisors and issued to the providers with recommendations.
Program Attendance Records and Waitlists
To assess whether DESE ensured that AE providers maintained accurate, complete, and up-to-date attendance records in the Literacy, Adult and Community Education System (LACES), we examined a statistical sample of 60 courses from a population of 1,828. Using LACES, we extracted provider-entered attendance records for those 60 courses. The courses were distributed among 41 providers. We also obtained the source attendance records from the 41 providers and compared them to the records maintained in LACES.
To assess whether AE providers maintained accurate, complete, and up-to-date waitlists, we evaluated DESE’s waitlist management processes and the level of oversight and guidance it gave to its providers. We selected a judgmental sample of 32 of 41 providers for interviews. During these interviews, we asked about procedures used to maintain waitlists for the providers’ AE courses. To arrive at our sample of 32, we excluded 5 providers that were not required to maintain waitlists because they were correctional facilities; 2 providers that no longer participated in DESE’s AE program; and 2 secondary locations for two community colleges, since their waitlist procedures were centralized.
To determine whether DESE monitored the management of AE providers’ waitlists, we reviewed the data quality checklists submitted by all 77 providers that were active during our audit period. The checklists serve as evidence that required data management activities were performed during the audit period, including those associated with waitlist maintenance.
Data Reliability
To assess the reliability of the attendance and waitlist data in LACES, we interviewed DESE management and staff members. To determine the completeness and accuracy of the data, we reviewed various attributes in the waitlist and attendance records in LACES (e.g., provider names, student names, course names, dates, and hours attended). Based on this review, we determined the LACES data to be sufficiently reliable for the purposes of this audit.
To assess the completeness and accuracy of the cohort list, we compared the AE providers on the cohort list to DESE’s list of funded AE programs. We determined that the cohort list was sufficiently reliable for the purposes of this audit.
To assess the reliability of the fiscal year 2019 AE provider cohort assignments, we compared each of the 77 provider RFP application scores from the cohort list to each provider’s fiscal year 2019 RFP application scorecard. We determined that the fiscal year 2019 AE provider cohort assignments from LACES were sufficiently reliable for the purpose of this audit.
Where nonstatistical sampling was used, we could not project the results of our testing to the overall populations.
| Date published: | June 2, 2020 |
|---|