Audit of the Collaborative for Educational Services Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Collaborative for Educational Services.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Collaborative for Educational Services (CES) for the period July 1, 2017 through March 31, 2019. Our testing of CES’s educator evaluation process covered the period September 1, 2017 through August 31, 2018 (the 2017–2018 school year), because that was the only complete evaluation cycle that occurred during our audit period.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and where each objective is discussed in the audit findings.

Objective	Conclusion
Did CES educators and teachers have licenses issued by the Department of Elementary and Secondary Education (DESE) as required by Section 4E of Chapter 40 of the General Laws?	No; see Finding 1
Did CES educators’ and teachers’ credentials correspond to their contracts, or positions, as required by Section 4E of Chapter 40 of the General Laws and Section 7.15(9)(a) of Title 603 of the Code of Massachusetts Regulations (CMR)?	No; see Finding 1
Were evaluations conducted in accordance with 603 CMR 35, CES’s Teacher Evaluation System guide, and Article 15 of the collective bargaining agreement between CES and the Service Employees International Union / American Federation of Labor and Congress of Industrial Organizations Local 509?	No; see Findings 2 and 3

 

To achieve our objectives, we gained an understanding of CES’s internal control environment related to the objectives by reviewing applicable agency policies and procedures, as well as conducting inquiries with CES management and personnel.

In addition, we performed the following audit procedures to obtain sufficient audit evidence to address our audit objectives.

Educator Licenses

We obtained from CES a payroll report generated from Infinite Visions² that included the names, job titles, and positions of all 971 individuals who worked for the agency during the audit period. We reviewed this list and, based on each individual’s position and job title (e.g., special education teacher), filtered the population to identify the 399 CES educators whom we expected to have educator licenses. We then selected a random, nonstatistical sample of 60 of these 399 employees for our substantive testing. We requested and obtained from CES the Educator Licensure and Renewal (ELAR)³ Inquiry—Account Summary page for each educator in our sample. We determined the dates each educator in our sample was employed at CES using a supplementary report provided by CES that detailed dates of employment. We compared the license information on the ELAR Inquiry—Account Summary page (such as activity; credentials; and application, issue, and expiration dates) to the information in the supplementary report and determined whether the educators in our sample had licenses or had received waivers that were valid during the audit period.

Educator Qualifications

We obtained the Education Personnel Information Management System⁴ reports submitted by CES to DESE during the audit period and reviewed the reported information for the same 60 CES employees we had randomly selected for our audit testing related to teacher licensure. We also reviewed the employment contracts of educators who had employment contracts, as well as their licensure information as it appeared on the ELAR Inquiry—Account Summary page. We reviewed the qualifications listed on the job descriptions for positions held by employees who did not have employment contracts. We compared educators’ employment contracts and/or job descriptions to their educator licenses to determine whether they had the appropriate licenses for the positions for which they were hired.

In addition, to obtain an understanding of the amount of time CES educators might teach subjects for which they might not be fully qualified (referred to as out-of-field teaching), we conducted an online survey of 159 educators who were employed by CES at the time the survey was conducted. We received responses from 124. We reviewed and analyzed the survey results, which we discuss in the “Other Matters” section of this report.

Educator Evaluations

We obtained the population of educators in TeachPoint and identified the 277 educators we determined were required to receive evaluations during the 2017–2018 school year. We selected a random, nonstatistical sample of 60 educators. We reviewed the evaluation forms in TeachPoint for each sampled educator to identify the primary evaluators, and/or their designees, who conducted the evaluations. We obtained and reviewed the ELAR Inquiry—Account Summary page for each primary evaluator identified to determine whether each one had the required administrator license to perform such evaluations.

We reviewed all evaluation forms in TeachPoint for each sampled educator to determine whether all required fields in each form were correctly completed; whether the completion dates corresponded to when the fields were to be completed; and that each evaluation was reviewed or signed by the educator, the evaluator, or both, as applicable.

We obtained and reviewed copies of the written authorizations from the appropriate supervisors to determine whether the teaching coordinators we identified as having performed evaluation activities (e.g., performing observations, reviewing goals, and writing evaluations) for the 60 sampled individuals had received the required authorization to do so. We also reviewed the educator plan assigned to each teaching coordinator who conducted teacher evaluations to determine whether the teaching coordinator was identified as being on an improvement plan.⁵ If a teaching coordinator was identified as being on an improvement plan, we requested the written authorization from the appropriate contract administrator for the teaching coordinator to conduct evaluation activities.

We used nonstatistical sampling methods and therefore did not project the results of our testing to the population.

Data Reliability Assessment

We assessed the reliability of the data obtained by (1) performing electronic testing by reviewing the data for blank fields, duplicate entries, and values outside the audit period; (2) reviewing CES policies and system controls; and (3) interviewing CES employees who were knowledgeable about the data. Additionally, we traced a random sample of 60 employees’ data from Infinite Visions to TeachPoint and traced a random sample of 60 employees’ data from TeachPoint to Infinite Visions. We determined that the data were sufficiently reliable for the purposes of this report.

2.     Infinite Visions is a third-party database owned by Tyler Technologies that CES uses to record human resources and financial data.

3.     The ELAR Portal is DESE’s online educator licensure system, which lists educators’ licenses and waivers, as well as the status of each.

4.     According to DESE’s website, this system “collects demographic data and work assignment information on individual public school educators.” This information is provided by schools.

5.      According to CES’s Teacher Evaluation System guide, an improvement plan is “a plan of at least 30 days and no more than one school year, for educators who are rated Unsatisfactory or Needs Improvement in any Standard or indicator, or overall, as determined by the Evaluator, developed by the Evaluator with goals specific to improving the educator’s Unsatisfactory performance. An Evaluator may place an educator on an Improvement Plan at any time.”

Date published:	July 22, 2021