Audit of the Massachusetts Bay Transportation Authority - Block by Block Contract Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Massachusetts Bay Transportation Authority - Block by Block Contract

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Massachusetts Bay Transportation Authority (MBTA) for the period January 1, 2021 through December 31, 2022.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Findings
1. To what extent did the MBTA ensure that Block by Block (BBB) employees were trained on MBTA operations and safety before being assigned to a subway station location in accordance with Sections 4.2.1 and 6.2.2 of the MBTA’s Request for Proposal No. 41-17 for In-Station Customer Service and Section 2.7 in the Scope of Work document of the MBTA’s Request for Proposal No. 3-22 for In-Station Customer Service Operations?	See Finding 1
2. Did the MBTA ensure that BBB completed visual inspections (station checks) of its subway station locations at the agreed-upon frequency?	See Finding 2

To accomplish our audit objectives, we gained an understanding of the MBTA’s internal control environment relevant to our objectives by reviewing applicable policies and procedures and by interviewing officials at the MBTA, the Massachusetts Department of Transportation, and BBB.

While reviewing the MBTA’s internal control environment, we found that the MBTA did not implement controls to establish accountability or effectively monitor BBB’s use of fare access cards. See the “Other Matters” section of this report for more information.

To obtain sufficient, appropriate evidence to address our audit objectives, we performed the following procedures.

BBB Employee Initial Training

To determine to what extent the MBTA ensured that BBB employees were trained on MBTA operations and safety before being assigned to a subway station location in accordance with Sections 4.2.1 and 6.2.2 (see Appendix A) of the MBTA’s Request for Proposal No. 41-17 for In-Station Customer Service  and Section 2.7 (see Appendix B) in the Scope of Work document of the MBTA’s Request for Proposal No. 3-22 for In-Station Customer Service Operations, we took the following actions.

We interviewed the MBTA’s deputy director of customer experience training, the MBTA’s senior director of the customer experience department, and BBB’s director of operations for MBTA ambassador services to discuss the BBB employee initial training program that was in place during the audit period. BBB provided us with a list of all 826 BBB employees who were active during the audit period.⁸ We then selected a random, nonstatistical sample of 60 of these BBB employees for testing. We reviewed documentation for each BBB employee’s initial subway station location assignment date and attendance sheets corroborating when they completed operations and safety trainings to determine whether these BBB employees had completed the trainings before being assigned to a subway station location.

We used nonstatistical sampling methods for testing and therefore did not project the results of our testing to the corresponding population.

See Finding 1 for information regarding an issue we identified related to untimely completion and non-completion of initial training for BBB employees. 

Station Checks

To determine whether the MBTA monitored whether visual inspections of its stations (station checks) were completed at the agreed-upon frequency of at least twice per hour, we took the following actions.

We reviewed the MBTA’s Requests for Proposal No. 41-17 for In-Station Customer Service and No. 3-22 for In-Station Customer Service Operations and other MBTA and BBB contract documents that were in effect during the audit period. We interviewed the MBTA’s chief procurement and contracts officer, manager of vendor contracts, and senior director of the customer experience department and BBB’s director of operations for MBTA ambassador services to discuss the services provided by BBB and the MBTA’s process for monitoring these services.

BBB provided us with a list of all station checks performed by BBB employees during the audit period from BBB’s Statistics Management and Ambassador Reporting Technology (SMART) system. Using the contract documents, we then identified all subway station locations⁹ covered by BBB during the audit period, including the number of hours¹⁰ that BBB employees were scheduled to work at these locations for each day during the audit period. We then analyzed the station check data obtained from BBB’s SMART system to determine whether BBB performed station checks at least twice per hour during contracted hours at every subway station location each day during the audit period.

See Finding 2 for information regarding an issue we identified related to the MBTA’s failure to ensure that BBB completed visual inspections (station checks) of its subway station locations at the agreed-upon frequency.

Data Reliability Assessment

BBB Employee Initial Training

To determine the reliability of the list of BBB employees who were employed during the audit period, which BBB provided to us, we checked the list for duplicate records and blank fields. We also ensured that the list did not include BBB employees with hire or termination dates outside of the audit period. To test the accuracy of the list, we judgmentally¹¹ selected a sample of 20 BBB employees from the list and compared these employees’ names and hire dates to source documentation (e.g., personnel files) provided by BBB. To test the completeness of the list, we judgmentally selected a different set of 20 BBB employees from source documentation at the BBB office (e.g., personnel files) and traced these back to the list of BBB employees who were employed during the audit period.

Station Checks

To determine the reliability of the list of all station checks performed by BBB employees during the audit period, which we obtained from BBB’s SMART system, we interviewed knowledgeable BBB employees about the system and the data in it. We also reviewed the SMART system data catalog to gain an understanding of the tables and fields that were available within the system. Further, we checked the list for duplicate records, blank fields, invalid or duplicate identifiers, and dates outside of the audit period. We also visited 10 MBTA subway station locations and observed BBB employees perform and document 20 station checks in the SMART system. We compared the subway station locations, and the date and time of the station checks, from our observation records to the data collected in the SMART system.

Based on the results of the data reliability assessment procedures described above, we determined that the information obtained was sufficiently reliable for the purposes of our audit.

8.    This list of BBB employees includes employees who were active at any point during the audit period.

9.    Each MBTA subway station location had a different number of days each week that BBB was contracted to staff the station. For example, during the audit period, the MBTA had BBB staff some subway station locations every day, other locations only on weekdays, and some locations not at all. We calculated the total number of days for each subway station location and combined those totals to determine that BBB had a combined 43,727 contracted days of staffing MBTA subway station locations during the audit period.

10.    Hours of BBB coverage varied by subway station location during the audit period. BBB staffed agreed-upon subway station locations with full coverage Monday through Saturday from 6:00 a.m. to 12:00 a.m. and Sunday from 7:00 a.m. to 12:00 a.m. We calculated the total number of BBB coverage hours for each day at each subway station location and combined those totals to determine that BBB had a combined 672,589 contracted hours of staffing MBTA subway station locations during the audit period.

11.    Auditors use judgmental (i.e., nonstatistical) sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review. Auditors use their knowledge and judgment to select the most appropriate sample. For example, an auditor might select items from areas of high risk. The results of testing using judgmental sampling cannot be used to make conclusions or projections about entire populations; however, they can be used to identify specific issues, risks, or weaknesses.

Date published:	July 7, 2024