Overview
The Department of Elementary and Secondary Education (DESE), established by Section 1A of Chapter 69 of the Massachusetts General Laws, is within the Commonwealth’s Executive Office of Education, under the supervision and management of the Commissioner of Elementary and Secondary Education. DESE is responsible for overseeing the education of children in pre-kindergarten through grade 12 in Massachusetts.
As of June 30, 2019, DESE had a total of 414 employees at its headquarters in Malden. For our audit period, DESE had state appropriations of approximately $5.6 billion and federal appropriations of approximately $1 billion. According to its website,
The Department of Elementary and Secondary Education provides leadership, oversight, funding, support, and accountability for the Commonwealth’s 404 school districts that are charged with educating close to 1 million public school children and 20,000 adult learners each year.
DESE’s Adult and Community Learning Services (ACLS) Unit is charged with managing the Commonwealth’s adult education (AE) program. The AE program was established under Section 1H of Chapter 69 of the General Laws to deliver adult basic education and literacy services intended to increase options for those requiring these services and lead to better employment opportunities. This section states,
Trainees shall include parents of young children who need strong basic skills to move their families out of poverty and raise the educational aspirations of their children.
The program offers educational assistance to Massachusetts residents who are 16 years and older, are not enrolled in high school, and require instruction ranging from basic literacy and numeracy, as well as English for speakers of other languages, to high school equivalency and college and career readiness skills. The services are delivered by third-party providers at no cost to the students.
The AE program is funded by both state and federal grants administered by DESE. During our audit period, DESE granted $39,927,932 to AE providers; $30,019,765 of this consisted of state appropriations and $9,908,167 consisted of federal appropriations.
AE Providers
The AE program in Massachusetts is delivered by independent providers through community-based organizations that offer adults easier ways to enroll in courses that are geographically accessible and that meet their educational needs.
AE providers are located in cities and towns throughout the Commonwealth. Providers include nonprofit community-based organizations, religious institutions, local governments (through their school systems), and correctional facilities. Information about the program’s existence is spread through word of mouth and referrals from other agencies.
AE Program Databases
In 1998, as the need for increased AE program performance accountability grew, DESE implemented a Web-based system, the System for Managing Accountability and Results Through Technology (SMARTT), which enabled AE providers to enter student performance data in a central database. DESE could now better monitor AE providers and identify what was going well and what could be improved in a more timely fashion. In fiscal year 2019, DESE replaced SMARTT with its latest software—the Literacy, Adult and Community Education System (LACES)—which providers could continue to use to capture student performance data. The new software also gave DESE additional reporting capabilities. DESE takes the details from LACES and enters key AE student performance data in the federal National Reporting System.1 Provider performance is a crucial component in receiving and maintaining AE program funding, so the reliability of the provider data entered in LACES is paramount in evaluating providers’ performance.
AE Provider Cohort Allocation and Frequency of Monitoring Visits
DESE grants funds to providers that have gone through an open and competitive bid process and have submitted applications in response to a request for proposals (RFP) from DESE (herein referred to as “RFP applications”). DESE reviews and scores the submitted applications. DESE then ranks the providers according to their RFP application scores,2 and because limited funds are available, it awards funds to those with the highest scores. This scoring system is also used to assign providers into three cohorts3 for oversight purposes. For fiscal year 2019, DESE granted funds to 77 AE providers. It assigned 25 providers to cohort 1, 26 providers to cohort 2, and 26 providers to cohort 3. Monitoring visits are scheduled based on these cohorts, as described below.
AE Provider Monitoring Process
DESE has four ACLS program specialists, including one program team leader, who make up the program quality review (PQR) team. They are responsible for monitoring providers’ performance and the quality of the curriculum delivered to adult learners. In addition to the program specialists, the PQR team uses other DESE staff members and outside consultants who have knowledge about specific monitored providers, as needed.
The provider monitoring process covers the four-year funding cycle for providers. Each provider is monitored by either a PQR or a site visit (SV), conducted by the PQR team, once a year over the first three years of the four-year funding cycle. No providers are reviewed in the fourth year, so that the PQR team can evaluate providers’ performance and determine grant allocations for the next four-year cycle. A PQR is a full-day monitoring visit. An SV is a half-day monitoring visit. A provider receives one PQR and two SVs over the four-year cycle.
DESE instituted this process in fiscal year 2019 to monitor the performance of AE providers and ensure that they complied with DESE policies for providing quality education to adult learners. In prior funding cycles, the PQR team reviewed only a sample of the providers. Although this gave the ACLS program specialists the option of conducting more thorough evaluations, the PQR team determined that there was an unacceptable risk involved in not reviewing some providers. The previous reviews lasted a few days, and the PQR team had time to go through all of the providers’ records and cross-reference data. This new protocol enables the team to monitor every provider during each of the first three years of the funding cycle.
According to DESE’s “Program Quality Review and Site Visit Protocol,” the team conducts these reviews for the following purposes:
1. to determine the quality of services against a set of Indicators through a diagnostic assessment (program quality reviews);
2. to offer technical assistance for program improvement (site visits).
3. to identify and disseminate promising practices that may impact student outcomes (program quality reviews and site visits).
4. to ensure state and federally-funded adult education providers are compliant with state and federal policies (program quality reviews and site visits).
The protocol further states that the goals of SVs are as follows:
1. Provide programs the opportunity to receive technical assistance (TA) grounded in and guided by the ACLS Indicators of Program Quality. Areas needing technical assistance may be identified in a number of ways:
a. Results of a self-assessment conducted by the program
b. Results of a desk review conducted by program specialist
c. Findings from the program quality review
d. Improvement plan triggered by the Program Accountability Protocol and/or open and competitive award funding conditions.
2. Provide ACLS with a deeper, more holistic understanding of the programs in order to identify areas of non-compliance and promising practices.
See the Appendix to this report for further information about indicators of program quality (IPQs).4 The process for both a PQR and an SV consists of a planning phase, a physical visit phase, and a report writing phase. The planning phase involves identifying the providers to visit in the coming year, announcing the visits, giving providers’ program directors general information about the visits, and requesting information and documents. Requested information and documents include scope and sequence of course instruction, lesson plans and curriculum for the visit date, resumes and job descriptions for the program’s leaders, and staff evaluations.
PQR and SV Processes
On the visit date, for both PQRs and SVs, the PQR team meets at the provider location. Pairs of team members evaluate the providers’ teaching skills and program leadership. They observe courses; tour the facility; and interview students, teachers, and program directors.
For PQRs, classroom observations are used to evaluate the level of curriculum rigor and the scope and sequence of instruction. The program leaders, students, and teachers are all asked the same set of questions, which are designed to address IPQs, as well as program goals, advisor availability, course study topics, and training programs. Program leader interviews last one and a half hours, student interviews last half an hour, and teacher and advisor interviews last one hour. For SVs, the agenda includes discussions with the program director regarding technical assistance, classroom observations, interviews with teachers, and wrap-up and recommendations for next steps with the program director.
The interview questions for PQRs focus on 4 of 10 predefined IPQs. To review all providers during the 2019–2022 funding cycle, the PQR team determined that PQRs should focus on IPQs 3, 4, 7, and 8, which are pathway collaborations (collaborations with local education and workforce partners designed to assist students with their next steps after leaving the program), curriculum and instruction, organizational capacity, and educational leadership, respectively. The interview questions for SVs focus on areas identified by an ACLS program specialist and the program director in the planning stage and on any findings that were revealed when the ACLS program specialist performed a prior desk review.
At the end of a PQR or an SV, the PQR team reassembles for a meeting to discuss its observations and the aggregated rating of each pair of team members. The group then comes to a consensus on the final rating and discusses any areas of contention. All of the notes are given to the team member who has been selected as the report writer.
Once the report is written and finalized, the provider is given an opportunity to respond, and any response is added to the report. The PQR and SV reports and responses are then submitted to the ACLS director for final approval.
| Date published: | June 2, 2020 |
|---|