Audit of the CPCS Objectives, Scope, and Methodology

An explanation of what the CPCS audit examined and how it was conducted.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Committee for Public Counsel Services (CPCS) for the period July 1, 2014 through December 31, 2016.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.  

Objective	Conclusion
Has CPCS established an effective audit and oversight (A&O) process in accordance with Section 12 of Chapter 211D of the General Laws?	Yes; see Other Matters
Does CPCS management verify work performed and approve vendor billings in compliance with established policies and procedures?	No; see Finding 1
Has the Office of the Commissioner of Probation (OCP) established policies and procedures to ensure that indigency eligibility and reassessment comply with Chapter 211D of the General Laws?	Partially; see Finding 2

 

To achieve our audit objectives, we gained an understanding of CPCS and its environment, evaluated the design of internal controls, and tested the effectiveness of internal controls over (1) processes related to A&O and (2) approval of vendor billings. We conducted further audit testing as described below.

A&O Department

Under Section 12 of Chapter 211D of the General Laws, CPCS’s A&O Department is responsible for monitoring the billings of private attorneys who accept cases from indigent clients through CPCS and vendors who provide expert services to CPCS attorneys.

We selected a statistical sample of 47 out of 1,731 E-Bill audits performed by attorneys during our audit period to determine, with a 90% confidence level, whether CPCS had established an effective A&O process. We reviewed E-Bill notices, bills submitted by attorneys, documentation of time records, proofs of insurance, copies of correspondence from A&O to attorneys, Notice of Assignment of Counsel (NAC) information reports, caseload reports, client reports, and processed bill reports for each E-Bill audit.

We selected a random nonstatistical sample of 25 expert vendor billings out of 295 that were audited by A&O during the audit period to verify compliance with A&O’s policies and procedures. We reviewed the selected vendor billings, supporting documentation, and court motions. In addition, we verified that vendors had completed and submitted V-Bill Access Rights Agreement Forms before submitting their first electronic bills as required by CPCS’s Court Cost Vendor Manual.

We reviewed CPCS’s billing data by performing statistical analysis using Microsoft Excel and applying data analytics using Audit Command Language to identify possible outliers within the data. We performed this analysis to determine whether alternative analytical approaches to analyzing this billing information could effectively be applied. In conducting this analysis, it was necessary to establish a norm (an expected or typical measurement value) and then identify outliers from this norm estimate. We defined “outlier” as any value outside the expected range of the other data points in our set of data using standard statistical sampling methods. We computed the norm based on observations from large samples of CPCS billing data.

We obtained CPCS attorney and vendor data from the Commonwealth Information Warehouse.³ We reviewed amounts paid to vendors for each fiscal year and performed an overall analysis of billing patterns for private attorneys during the audit period.

OCP

In a prior audit (No. 2011-1104-3C), we noted significant deficiencies with the process of verifying indigency and provided recommendations to OCP as well as the Administrative Office of the Trial Court on how to improve this process. During our current audit, to determine what measures, if any, had been taken to address these deficiencies, we performed the work described below.

We reviewed Section 2A of Chapter 211D of the General Laws and Supreme Judicial Court Rule 3:10 to gain an understanding of the requirements of the Chief Probation Officers (CPOs) at individual courts.

There are 109 criminal courts in the Commonwealth, including 27 juvenile courts, 70 district courts, and 12 superior courts. During our audit period, defendants were deemed indigent by the court and assigned counsel by CPCS in 336,461 cases. Because indigency records are located throughout the Commonwealth, we selected a judgmental sample of eight court locations (two juvenile courts, four district courts, and two superior courts) to test for compliance with the Massachusetts Probation Services’ (MPS’s) Indigency Verification Protocols. We randomly selected five NACs from each court for the period September 1, 2014 through December 31, 2016. We then obtained the docket numbers associated with these NACs. We met with each of the eight CPOs and determined who performed the verifications of indigency and reassessments. We reviewed the supporting documentation, including the required MPS forms, authorizations, and corroborating evidence of indigency.

In addition, because of an update in the indigency verification process with the release of the MPS Indigency Verification Policy dated November 1, 2016, we judgmentally selected eight additional courts (two juvenile courts, four district courts, and two superior courts). We selected five NACs from each court for the period December 12, 2016 through December 31, 2016 to test the indigency verification process under the new policy. This review required MPS to provide case files and supporting information to the audit team without our conducting site visits.

Data Reliability

We obtained data from CPCS’s AccuTerm database, which maintains information on private attorneys’ billable time and expenses by service date. In addition, we obtained three spreadsheets listing the overall number of billings, including the total dollar amount paid by CPCS for each service type. We reviewed controls for access to programs and data, program changes, and computer operations. We also performed a data reliability assessment by performing additional validity and integrity testing, including (1) testing for missing data, (2) scanning for duplicate records, (3) tracing a sample of cases queried to source documents, and (4) reviewing spreadsheets for hidden or erroneous content. Our data reliability assessment of CPCS data included completeness and accuracy testing. We determined that the data were sufficiently reliable for the purposes of our audit testing.

Except where noted, we used nonstatistical sampling to help us achieve our audit objectives and therefore did not project our results to the various populations.

3. According to the website of the Executive Office of Technology Services and Security, the Commonwealth Information Warehouse is a repository of “financial, budgetary, human resource, payroll, and time reporting information.”

Date published:	February 28, 2018