Audit of the Brockton Area Transit Authority Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Brockton Area Transit Authority.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Brockton Area Transit Authority (BAT) for the period January 1, 2021 through December 31, 2022.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer and the conclusion we reached regarding each objective.

Objective	Conclusion
1. Does BAT deliver paratransit services required by the Americans with Disabilities Act (ADA) on time in accordance with Section 37.23(a) of Title 49 of the Code of Federal Regulations (CFR); Sections 5.3, 5.4, and 5.6 of Chapter 8 of the Federal Transit Administration’s Circular 4710.1; and BAT’s “ADA Paratransit Services Guide”?	Yes
2. Does BAT ensure that all ADA paratransit complaints are investigated and responded to as required by 49 CFR 27.13(b) and BAT’s “ADA Paratransit Services Guide”?	Yes

To accomplish our audit objectives, we gained an understanding of the aspects of BAT’s internal control environment relevant to our objectives by reviewing applicable agency policies and procedures and by interviewing BAT’s employees and management. We evaluated the design of controls over ADA-required paratransit services and ADA paratransit complaints.

To obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Performance Standards for ADA-Required Paratransit Services

To determine whether BAT delivered ADA-required paratransit services required on time in accordance with 49 CFR 37.23(a); Sections 5.3, 5.4, and 5.6 of Chapter 8 of the Federal Transit Administration’s Circular 4710.1; and BAT’s “ADA Paratransit Services Guide,” we took the actions described below.

We obtained a dataset of all trips that occurred during the audit period from BAT. From this dataset, we extracted the following:

• all 39,228 ADA-required paratransit trips that were both on time and untimely (which we compiled into a new list); and
• all 8 ADA-required paratransit trips that were considered missed trips.

To determine whether BAT met the 85% performance standard it established with its contracted paratransit provider and the Federal Transit Administration’s Circular 4710.1, we analyzed 100% of the 39,228 ADA-required paratransit trips using Microsoft Excel. To determine whether BAT met its 85% internal performance standard, we analyzed 100% of the 8 ADA-required paratransit trips that were considered missed trips. We compared the information corresponding to the 39,228 ADA-required paratransit trips and the 8 missed trips to BAT’s performance standards regarding on-time performance, untimely drop-offs, and missed trips, the process of which we detail below.

On-Time Performance

To determine whether BAT complied with its performance standard of picking up at least 85% of riders within 15 minutes after the scheduled pickup time, we took the following actions. We calculated the number of minutes between the ADA-required paratransit rider’s actual pickup time and scheduled pickup time for each of the 39,228 trips that occurred during the audit period. We calculated the percentage of pickups that were on time from every trip that occurred during the audit period and compared this percentage (88%) to BAT’s performance standard to determine whether BAT met its on-time performance standard. To determine whether BAT met its performance standard during the audit period, we compared the percentage of pickups that were on time (88%) to the performance standard in BAT’s memorandum of understanding with the Massachusetts Department of Transportation. We identified 4,465 late pickups that were outside of the on-time pickup window and sorted these trips by the number of minutes that the driver was late. We then targeted the 109 trips that had excessively late pickups (which we defined as being 71 minutes or more past the scheduled pickup time). For the remaining 4,356 late pickups, we organized them into the following intervals of minutes past the scheduled pickup times: 1–15 minutes, 16–30 minutes, 31–45 minutes, 46–60 minutes, and 61‍–‍70 minutes. Then, from each interval, we selected a number of late pickups that was proportional to each interval’s percentage in relation to the total number of late pickups. These selections became our random, statistical⁴ sample of 47 trips, which we formed using a 90% confidence level,⁵ a 0% expected error rate,⁶ and a 10% tolerable error rate.⁷ For both of the samples (the targeted 109 trips with excessively late pickups and the randomly selected 47 trips with late pickups), we examined the manager of transit operations’s monthly reconciliation notes corresponding to late pickups, and we identified the cause of each late pickup from these notes.

Untimely Drop-offs

To determine whether BAT complied with its performance standard of having an untimely drop-off rate of no more than 15% of all scheduled drop-offs, we took the following actions:

• We calculated the number of minutes between the ADA-required paratransit rider’s actual drop-off time and the scheduled drop-off time for each of the 39,228 trips that occurred during the audit period.
• We calculated the percentage of drop-offs that were untimely and compared this percentage (10.84%) to BAT’s performance standard to determine whether BAT met its standard for untimely drop-offs.
• We then targeted 10 untimely drop-offs that were excessively late (which we defined as being 71 minutes or more past the scheduled drop-off time) from the total population of 1,228 untimely drop-offs that occurred during the audit period.
• For the remaining 1,218 untimely drop-offs, we organized them into the following intervals of minutes past the scheduled drop-off times: 1–15 minutes, 16‍–‍30 minutes, 31–45 minutes, 46–60 minutes, and 61–70 minutes.
• From each of the above intervals, we selected a number of untimely drop-offs that was proportional to each interval’s percentage in relation to the total number of untimely drop-offs. These selections became our random, statistical sample of 47 trips, which we formed using a 90% confidence level, a 0% expected error rate, and a 10% tolerable error rate.
• For both of the samples (the targeted 10 untimely drop-offs that were excessively late and the randomly selected 47 trips with untimely drop-offs), we examined the manager of transit operations’s monthly reconciliation notes corresponding to untimely drop-offs, and we identified the cause of each untimely drop-off from these notes.

Missed Trips

To determine whether BAT complied with its performance standard of having a missed trip rate of no more than 2% of all scheduled trips, we took the following actions: We assessed 100% of all eight trips that were missed during the audit period by calculating the percentage of missed trips from the total amount of trips scheduled during the audit period. We then examined the manager of transit operations’s monthly reconciliation notes corresponding to all eight missed trips, and we identified the cause of each missed trip from these notes.

We noted no exceptions in our testing; therefore, we conclude that, during the audit period, BAT delivered ADA-required paratransit services on time in accordance with 49 CFR 37.23(a); Sections 5.3, 5.4, and 5.6 of Chapter 8 of the Federal Transit Administration’s Circular 4710.1; and BAT’s “ADA Paratransit Services Guide.”

ADA Paratransit Complaints

To determine whether BAT ensured that all ADA paratransit complaints were investigated and responded to as required by 49 CFR 27.13(b) and BAT’s “ADA Paratransit Services Guide,” we took the following actions. We obtained a dataset of ADA paratransit complaints from BAT. We analyzed 100% of the 39 ADA paratransit complaints that BAT received during the audit period. We calculated the business days between the date BAT received each complaint and the date of BAT’s documented follow-up with the complainant, which included the investigation of and the resolution to the ADA paratransit complaint. We identified 9 complaints that were not resolved within 72 hours and requested additional information for these 9 complaints to determine why BAT did not resolve these complaints within 72 hours. We examined the information corresponding to each record for these 9 ADA paratransit complaints (i.e., the description of the complaint and BAT’s investigation comments regarding it), and cross-referenced this information to the additional notes for each complaint that BAT’s manager of transit operations provided to us. We performed this cross-reference to determine the date of the resolution and the reason why the complaint was not resolved within 72 hours.

We noted no exceptions in our testing; therefore, we conclude that, during the audit period, BAT ensured that all ADA paratransit complaints are investigated and responded to as required by 49 CFR 27.13(b) and BAT’s “ADA Paratransit Services Guide.”

Data Reliability Assessment

Transit Scheduling and Dispatching System

To determine the reliability of the ADA-required paratransit trip dataset from BAT’s transit scheduling and dispatching system, we interviewed officials who were knowledgeable about the ADA-required paratransit trip dataset and tested selected information system controls (e.g., security management, configuration management, contingency planning, and segregation of duties) corresponding to this data. We then tested the ADA-required paratransit trip dataset to ensure that there were no records that were missing, duplicate, or outside of the audit period.

We obtained a list of 351 riders who were on BAT’s list of approved ADA-required paratransit riders from BAT’s administrator, selected a random sample of 20 riders, and traced each of these 20 ADA-required paratransit rider records to hardcopy rider applications. We randomly selected 20 hardcopy ADA-required paratransit rider applications and traced these applications to the list of riders who were on BAT’s ADA-required paratransit service list. We verified that all riders who received ADA-required paratransit services from BAT during the audit period were on BAT’s list of riders eligible for ADA-required paratransit services.

When we found missing information within the ADA-required paratransit trip dataset, we followed up with management to obtain reasonable explanations for the missing data before conducting our analyses. While there was a small number of missing data points in BAT’s datasets—out of all the 39,228 trips that occurred during the audit period, 187 trips (0.17%) were missing a pickup time and out of the 11,682 trips with a scheduled drop-off time, 9 trips with a scheduled drop-off time (0.08%) were missing a drop-off time—this issue does not have a significant impact on BAT’s on-time performance and untimely drop-off metrics.

Complaint Management System

To determine the reliability of the ADA paratransit complaint dataset from BAT’s complaint management system, we interviewed officials who were knowledgeable about the datasets for ADA-required paratransit trips and ADA paratransit complaints. We tested selected information system controls (e.g., security management, configuration management, contingency planning, and segregation of duties) corresponding to the complaint management system. We tested the ADA paratransit complaint dataset to ensure that there were no records that were missing, duplicate, or outside of the audit period. When we found missing information within the ADA paratransit complaint dataset, we followed up with management to obtain reasonable explanations for the missing data before conducting our analyses.

Based on the results of the data reliability procedures described above, we determined that the information obtained for the audit period was sufficiently reliable for the purposes of our audit.

Conclusion

Our audit revealed no significant issues that must be reported under generally accepted government auditing standards. We noted no exceptions in our testing; therefore, we conclude that BAT delivered ADA-required paratransit services on time, and BAT ensured that all ADA paratransit complaints are investigated and responded to, during the audit period.

4.   Auditors use statistical sampling to select items for audit testing when a population is large (usually over 1,000) and contains similar items. Auditors generally use a statistics software program to choose a random sample when statistical sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

5.   Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

6.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

7.   The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

Date published:	July 24, 2024