Audit of the Berkshire Regional Transit Authority Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Berkshire Regional Transit Authority.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Berkshire Regional Transit Authority (BRTA) for the period January 1, 2021 through December 31, 2022.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
1. Does BRTA deliver paratransit services required by the Americans with Disabilities Act (ADA) on time in accordance with Section 37.23(a) of Title 49 of the Code of Federal Regulations; Sections 5.3, 5.4, and 5.6 of Chapter 8 of the Federal Transit Administration’s (FTA’s) Circular 4710.1; and BRTA’s “ADA Policies and Procedures”?	No; see Finding 1
2. Does BRTA ensure that all complaints from ADA-required paratransit riders are investigated and responded to as required by Section 27.13(b) of Title 49 of the Code of Federal Regulations and BRTA’s “ADA Policies and Procedures”?	No; see Finding 2

To accomplish our audit objectives, we gained an understanding of the aspects of BRTA’s internal control environment relevant to the objectives by reviewing applicable agency policies and procedures and by interviewing BRTA employees and management. We evaluated the design of controls over ADA-required paratransit services and ADA paratransit complaints.

To obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Performance Targets for ADA-Required Paratransit Services

We determined whether BRTA complied with its targets for ADA-required paratransit services regarding on-time performance, untimely drop-offs, and missed trips by taking the following actions:

• We obtained data from BRTA’s transit scheduling and dispatching system regarding all trips that occurred during the audit period and extracted the 34,638 ADA-required paratransit trips that occurred during the audit period.
• Using Microsoft Excel, we analyzed all of these 34,638 ADA-required paratransit trips to determine whether BRTA met the performance targets in BRTA’s “ADA Policies and Procedures” and FTA’s guidelines in FTA Circular 4710.1.
• We compared the information corresponding to these 34,638 trips to BRTA’s performance targets regarding on-time performance, untimely drop-offs, and missed trips by performing the testing described below.

On-Time Performance

To determine whether BRTA complied with the performance target of picking up at least 96.25% of riders within the scheduled pickup window, we took the following actions. We calculated the number of minutes between the ADA-required paratransit rider’s actual pickup time and the scheduled pickup time for each of the trips that occurred during the audit period. We calculated the percentage of pickups that were on time from every trip that occurred during the audit period and compared this percentage to the performance target established in the memorandum of understanding between BRTA and the Massachusetts Department of Transportation. We identified 5,370 late pickups that were outside of the on-time pickup window and sorted these trips by the number of minutes that they were late. We selected a random, statistical sample⁶ of 47 trips out of the 5,370 trips with late pickups using a 90% confidence level,⁷ a 0% expected error rate,⁸ and a 5% tolerable error rate.⁹ In addition, we interviewed BRTA officials about the reasons for the 5,370 late pickups that occurred during the audit period.

Untimely Drop-Offs

To determine whether BRTA met FTA’s performance target of having an untimely drop-off rate of no more than 0%, we took the following actions. We identified 8,454 trips that resulted in riders being dropped off after their scheduled drop-off time. We then calculated the number of minutes between each rider’s actual drop-off time and scheduled drop-off time for each of these trips. We selected a random, statistical sample of 47 trips out of the 8,454 untimely drop-off trips, using a 90% confidence level, a 0% expected error rate, and a 5% tolerable error rate. In addition, we met with BRTA officials to identify the cause of each untimely drop-off in our sample.

Missed Trips

To determine whether BRTA met FTA’s performance target of having a missed trip rate of no more than 0%, we took the following actions. We calculated the number of minutes between the actual pickup time and the scheduled pickup time for each of the ADA-required paratransit trips that occurred during the audit period. We identified 477 trips where the driver arrived more than 30 minutes after the scheduled pickup time. We determined that the percentage of missed trips for the audit period was 1.38%. From the 477 trips, we selected a random, nonstatistical sample¹⁰ of 40 trips out of the 477 missed trips by targeting 100% (27) of the trips where the driver was over two hours late and randomly selected 13 trips from the remaining 450 trips. In addition, we met with BRTA officials to identify the cause for each missed trip in our sample.

See Finding 1 for more information regarding the results of our testing regarding whether BRTA delivered ADA-required paratransit services on time and in accordance with its performance targets.

ADA Paratransit Complaints

To determine whether BRTA investigated and responded to ADA paratransit complaints, we took the following actions. We reviewed all of the 17 valid paratransit complaints that BRTA received during the audit period. Of these 17 paratransit complaints, one was related to ADA concerns. We reviewed the one ADA paratransit complaint to determine whether (1) BRTA’s contracted paratransit provider documented the complaint in a summary log, (2) the complainant received an acknowledgement of receipt from BRTA’s contracted paratransit provider customer service representative, and (3) BRTA notified the complainant of the determination of the investigation within seven days after the investigation was completed.

See Finding 2 for more information regarding the results of our testing of BRTA’s ADA paratransit complaint process.

We used a combination of statistical and nonstatistical sampling methods for testing, and we did not project the results of our testing to any populations.

Data Reliability Assessment

Transit Scheduling and Dispatching System

To determine the reliability of the ADA-required paratransit trip dataset from BRTA’s transit scheduling and dispatching system, we interviewed officials who were knowledgeable about the ADA-required paratransit rider data and tested selected information system controls (security management, configuration management, contingency planning, segregation of duties, and application controls). We then tested the ADA-required paratransit trip dataset to ensure that there were no records that were missing, duplicate, or outside of the audit period.

We obtained a list of 1,298 riders who were on BRTA’s list of approved ADA-required paratransit riders from BRTA’s administrator. To test this list for accuracy, we randomly selected a sample of 20 riders’ names and the dates of their trips from this list and compared these to their hardcopy ADA-required paratransit rider applications to confirm that the riders’ names and the dates of their trips matched. To test this list for completeness, we randomly selected 20 hardcopy ADA-required paratransit rider applications and traced these applications to the list. We verified that all riders who received ADA‑required paratransit services from BRTA during the audit period were on BRTA’s list of approved ADA‑required paratransit riders.

ADA Paratransit Complaint Documentation

To determine the reliability of the list of ADA paratransit complaints we received from BRTA’s electronic file storage, we interviewed officials who were knowledgeable about BRTA’s complaint records. We compared the total number of complaints from BRTA’s electronic file storage for calendar years 2021 and 2022 to the number of valid complaints published on BRTA’s website.

To determine the reliability of the complaint data, we obtained electronic copies of the 36 complaints that BRTA received during the audit period and identified 17 complaints that BRTA considered valid. We verified that there were no gaps in the sequential numbers used to identify each complaint. We also verified that there were no duplicate complaint records.

Based on the results of the data reliability assessment procedures described above, we determined that the information we obtained was sufficiently reliable for the purposes of our audit. 

6.    Auditors use statistical sampling to select items for audit testing when a population is large (usually over 1,000) and contains similar items. Auditors generally use a statistics software program to choose a random sample when statistical sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

7.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

8.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

9.    The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

10.    Auditors use nonstatistical sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review.

Date published:	October 4, 2024