I/O Solutions has audited and found that the test results from the Probation promotional exam administered on March 21, 2015, were valid and consistent with I/O Solutions' detailed scoring process. The work that was performed for the Massachusetts Trial Court by I/O Solutions with regard to test design, development and administration adhered to industry standards for employment-related testing. Current Probation employees who passed the exam are now eligible to apply for Assistant Chief and Chief Probation Officer Positions in the Commonwealth’s criminal courts.
As a result, the Massachusetts Trial Court has resumed the promotional process to fill critical vacancies in its Assistant Chief and Chief Probation Officer ranks in dozens of courts across the Commonwealth. As mandated in the 2011 Court Reform legislation, all applicants for new hire or promotional opportunities in either Trial Court Security or the Massachusetts Probation Service are required to take and pass a pre-qualifying examination before applying for any position. This is part of a larger effort to develop standards for appointments to critical positions in these Departments and to build and support the workforce across the Trial Court. The Trial Court contracted with I/O Solutions in 2014, a national test development and administration company with extensive experience in criminal justice and law enforcement testing, to develop the required hiring and promotional exams for both Court Security and the Probation Service.
The Probation promotional exam was administered on March 21, 2015 to fill vacancies in the adult criminal courts. Following the issuance of the test results, the union that represents probation officers and assistant chief probation officers raised concerns to the Trial Court. In response to the union's concerns and to ensure that the test results were valid and fair, the Trial Court asked I/O Solutions to conduct an audit to ensure that the test scoring process followed by the Trial Court met the detailed scoring process developed by I/O Solutions to meet industry best practices.
“Workforce development is an important goal for the Massachusetts Probation Service and the Trial Court, in general. We are mindful of the impact that recent hiring and promotional changes have had on our staff. As part of this development effort the Massachusetts Probation Service will offer trainings designed to provide its staff with the knowledge, skills and overall competencies needed to perform this important work," said Trial Court Administrator Harry Spence. "Employees who did not pass the March 21st exam will have future opportunities to take a promotion exam and are encouraged to take advantage of the training and other development opportunities provided.”
I/O Solution, in completing its review, stated its conclusions in a written report to the Trial Court. The I/O Solutions report states:
- The purpose of the audit was to evaluate the integrity of the rating process with regard to the accuracy and completeness of the raters’ documentation, adherence to IOS procedures, occurrences of unethical behavior or breaches of confidentiality and anonymity, and the influence of rater assignment on the passing or failing of candidates.
- As a result of the audit, IOS concluded that the rating process for the 2015 Massachusetts Trial Court, Probation Department promotional essay exam was fair, consistent, and a valid implementation of the procedures outlined by IOS.
- IOS also found the raters’ documentation to be complete and did not uncover any problematic incidents in the process that threaten the validity of the exam.
- Without reservation, IOS recommends that the Massachusetts Trial Court Human Resources department use the results of the 2015 promotional examination to create eligibility lists for the above positions.
The development of the exams was extensive and followed industry best practices. I/O Solutions conducted a content validation study to document the job-relatedness of the tests in accordance with the Federal Uniform Guidelines on Employee Selection Procedures. I/O Solutions worked with Trial Court staff extensively to produce a detailed job analysis process to identify the essential duties performed by these positions, and conducted numerous focus group meetings with a large number of employees and managers in these Departments to develop the exam content, collect validation evidence, develop a plan for the administration of the exams, and develop detailed scoring processes using objective criteria. The completion of the review and validation of the test results is an important step in fully implementing the new hiring, promotional and staff development procedures in the Probation Service.