In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Massachusetts Bay Transportation Authority (MBTA) for the period January 1, 2021 through December 31, 2022.
We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.
| Objective | Conclusion |
|---|---|
| No; see Findings 1 and 6 |
| No; see Finding 2 |
| No; see Finding 3 |
| No; see Finding 4 |
| To an unsatisfactory extent; see Finding 5 |
To accomplish our objectives, we gained an understanding of MBTA’s internal control environment relevant to our objectives by reviewing applicable policies and procedures and by interviewing officials at the MBTA.
To obtain sufficient, appropriate evidence to address our audit objectives, we performed the following procedures.
Scope Limitations
Section 9.12 of the US Government Accountability Office’s Government Auditing Standards states, “Auditors should . . . report any significant constraints imposed on the audit approach by information limitations or scope impairments.” During our audit of the MBTA, there were significant constraints on our ability to obtain the information necessary to achieve our objectives, as follows.
Subway Track Visual Inspections
To determine whether the MBTA ensured that visual inspections of its subway tracks were performed in accordance with Section 151.11(2)(a) of Title 220 of the Code of Massachusetts Regulation, we requested from MBTA management a list of all subway track visual inspections conducted during the audit period. The MBTA informed us that it could not provide the data for the period January 1, 2021 through April 4, 2021 because the MBTA’s enterprise asset management system, used to record subway track inspections, was only implemented in April 2021. According to MBTA officials, all records before this implementation were maintained manually, and retrieving these records would result in excessive delays to our audit work. As a result, we limited the scope of our audit for Objective 1 to the period April 5, 2021 through December 31, 2022.
We interviewed the MBTA’s director of maintenance of way (MOW), the senior director of MOW, the MBTA’s director of reliability and engineering configuration, and a section foreperson to discuss the process of visual inspections of subway tracks that was in place during the audit period. The MBTA provided us with an inventory list of all subway tracks by inspection segment as well as a list of all subway track inspections for the period April 5, 2021 through December 31, 2022 (a total of 6,373 track inspections). We then analyzed the data to ensure that a track inspector performed the inspection three times a week with at least one calendar day interval between inspections for light-rail subway track and twice weekly with at least one calendar day interval between inspections for heavy-rail subway track.
Based on the results of this analysis, we found that 4,309 of the PM work orders were not reviewed or were closed on the same day as the inspection. We then selected a sample of 60 inspections for further testing as follows. We first selected a random, nonstatistical sample of 30 inspections from the 4,309 PM work orders. We then selected an additional, judgmental sample of 30 inspections based on the longest duration between when the PM work order was opened and when it was closed. For each of the inspections in our sample of 60, we compared the date of actual inspection that was reported on the hardcopy Track Inspection Report to the scheduled date of the PM inspection found in the MBTA’s enterprise asset management system.
We noted that the MBTA did not ensure the timely completion of preventive maintenance work orders. See Finding 1 for information on the results of this testing.
Subway Track Defect Reporting and Remediation
To determine whether the MBTA ensured that the subway track defects identified during visual inspections were remediated within the requirements of the MBTA’s Track Maintenance Standards, we requested from MBTA management a list of all subway track defects reported during the audit period. We interviewed the MBTA’s director of MOW, the senior director of MOW, the MBTA’s director of reliability and engineering configuration, and a section foreperson to discuss the subway track defect reporting and remediation process that was in place during the audit period. The MBTA provided us with a list of 5,998 track service requests that detailed information and issues found during visual inspections during the audit period. The MBTA also provided us a list of 22,674 subway track defects for the period May 14, 2021 through December 31, 2022. MBTA officials informed us that they could not provide track defect verification data for the period January 1, 2021 through May 13, 2021 because the MBTA’s electronic defect module was not implemented until May 14, 2021. According to MBTA officials, before implementing the defect module, all records were maintained manually and retrieving them would have resulted in excessive delays to our audit work. As a result, our testing of the verification of existing defects was limited to the period May 14, 2021 through December 31, 2022. Lastly, the MBTA provided a mapping file that we used to determine which defect criteria to use for testing.
From the list of track asset service requests, we selected a random, statistical sample9 of 60 out of 5,998 track asset service requests, using a 95% confidence level, a 0% expected error rate,10 and a 5% tolerable error rate.11 For each track asset service request in our sample, we reviewed the priority level of each defect and determined whether the MBTA could substantiate that it had performed the remedial action required by the MBTA Track Maintenance Standards. Specifically, we determined whether a speed restriction, if required, was documented in the MBTA’s enterprise asset management system and whether the repairs were scheduled or completed within the required time frames. We also determined whether all existing defects were being verified and/or updated by track inspectors during subsequent inspections.
We noted that the MBTA did not ensure that defects identified during inspections of subway tracks were remediated in accordance with its Track Maintenance Standards. See Finding 2 for information on the results of this testing.
Subway Station Facility Inspection
To determine to what extent the MBTA performed visual inspections of its subway station facilities in accordance with Section 6.2.2 of its 2018 and 2022 Transit Asset Management Plans, we took the actions described below.
We interviewed the MBTA’s director of transit facilities maintenance to discuss the MBTA’s process for routinely inspecting its subway station facilities. Specifically, we discussed the various types of routine inspections performed, the frequency of such inspections, what each inspection entailed, how each inspection was documented, and the defect remediation process, including timelines. We requested documentation to substantiate that the MBTA performed these inspections for all 122 subway stations throughout the MBTA system during the audit period. In response to our request for documentation, the MBTA only provided annual inspection reports for 50 (41%) of the 122 stations for the year 2021. The MBTA could not demonstrate that any monthly or quarterly inspections were conducted for any stations in 2021. Furthermore, no visual inspections for subway stations could be substantiated for the year 2022. As a result, our review was limited to the 50 annual inspection documents that were made available to us for 2021. We inspected the 50 annual inspection reports for 2021 to determine when each inspection was performed and whether the reports included clear documentation of the inspection results, identifying any deficiencies and the remediation efforts taken to address those deficiencies.
We noted that the MBTA did not ensure that visual inspections of its subway station facilities were performed. See Finding 5 for information on the results of this testing.
Subway Car Inspection
To determine whether the MBTA ensured that PM inspections of its subway cars were performed in accordance with its Equipment Engineering and Quality Assurance Vehicle Inspection Guidelines we took the following actions:
- We interviewed the MBTA’s chief mechanical officer, the director of MOW, the deputy director of vehicle maintenance, and the project manager of transit applications to discuss the subway car PM inspection process that was in place during the audit period. The MBTA provided us with a list of all MBTA subway cars and all PM inspections for the audit period.
- During the period January 1, 2021 through December 31, 2022, we analyzed all 4,965 PM inspection work orders, comparing the mileage and day intervals between each scheduled inspection, to determine whether each subway car received an inspection in accordance with the intervals outlined in the MBTA’s Equipment Engineering and Quality Assurance Vehicle Inspection Guidelines.
We noted that the MBTA did not always perform preventive maintenance on its subway cars at the required intervals. See Finding 3 for information on the results of this testing.
Subway Car Defect Reporting and Remediation
To determine whether the MBTA ensured that subway car defects identified during PM inspections were remediated within the requirements of the MBTA’s Preventive Maintenance Inspection and Documentation Policy, we took the actions described below.
We interviewed the MBTA’s chief mechanical officer, the director of MOW, the deputy director of vehicle maintenance, and the project manager of transit applications to discuss the defect remediation process of its subway cars that was in place during the audit period. The MBTA provided us with a list of all 5,681 subway car PM work orders from the audit period. The chart below shows the average number of PM work orders per car, broken down by subway line, for calendar years 2021 and 2022.
Average Number of Work Orders per Car per Subway Line
From this list, we selected a random, statistical sample of 60 work orders, using a 95% confidence level, a 0% error rate, and a 5% tolerable error rate. Within this sample of 60 work orders, we identified 390 total deferred defects (averaging 6.5 deferred defects per work order, with a range of 1 to 19 defects per work order). We then performed the following tests:
- Based on the severity code of each reported defect, we determined whether MBTA performed prescribed remedial action based on the MBTA Preventive Maintenance Inspection and Documentation Policy as follows12:
- For Code A, we inspected the mileage report to determine whether the subway car was held from service until repaired.
- For Code B, we inspected asset service request and work order records to determine whether the subway car was inspected within 45 days from the date of PM.
- For Code C, we inspected asset service request and work order records to determine the date that the subway car was repaired, if at all.
We noted that the MBTA did not consistently document deferred defects found during subway car inspections and did not always accurately record their associated severity codes. See Finding 4 for information on the results of this testing.
We used a combination of statistical and nonstatistical sampling methods for testing, and we did not project the results of our testing to the corresponding populations.
Data Reliability Assessment
MBTA Enterprise Asset Management System
To determine the reliability of the data that we obtained from the MBTA’s enterprise asset management system for the audit period, we interviewed MBTA officials who were knowledgeable about the data. We also reviewed select system controls related to access controls, configuration management, contingency planning, segregation of duties, and security management. Through this testing, we found that the MBTA did not ensure that all its information system users completed cybersecurity awareness training. See Finding 6 for more information regarding the results of our review of the information system controls.
From the MBTA’s enterprise asset management system, we obtained a list of all subway track inspections for the period April 1, 2021 through December 31, 2022; a list of all subway track asset service requests and a list of all subway car inspections during the audit period. We also reviewed the enterprise asset management system work order interface data field mapping to gain an understanding of the tables and fields available within the system. For each of these data sets, we reviewed the parameters used in generating the list from the system. We also checked the list for missing data, blank fields, meaningful or expected values, and dates outside the audit period.
To determine the reliability of the list of subway track inspections, we judgmentally selected a sample of 20 track inspection work orders from the list and compared the work order number, work order location identification number, work order year, PM service track segment, date of PM schedule, and user identification number of the employee who closed the work order to the source document (inspection work orders). Further, we performed a reconciliation to match the data from the enterprise asset management system to the list of track inventory by inspection segment received from the MBTA’s track inspection and defect verification system.
To determine the reliability of the list of subway track asset service requests, we judgmentally selected 20 service requests from the list and compared the equipment number, status, line, date, priority, unique identification number, problem, and description to the original documents. Further, we performed a reconciliation to match the subway track asset service requests to the list of defect verifications received from the MBTA’s track inspection and defect verification system.
To determine the reliability of the list of subway car inspections, we judgmentally selected a sample of 20 subway car inspections from the list and compared the line, unit, labor date, mileage meter at the time, work order, task description, and employee identification number to the original documents, (inspection forms and work orders). Further, we judgmentally selected a sample of 20 subway car inspection forms and traced the line, unit, labor date, meter at the time, work order, task description, and employee identification number to the list of subway car inspections received from the MBTA’s enterprise asset management system.
MBTA Defect Module
To determine the reliability of the defect verifications received from the MBTA’s defect module for the period May 14, 2021 through December 31, 2022, we interviewed MBTA officials who were knowledgeable about the data. We also reviewed select system controls related to access controls, configuration management, contingency planning, segregation of duties, and security management. Through this testing, we found that the MBTA did not ensure that all its information system users completed cybersecurity awareness training. See Finding 6 for more information regarding the results of our review of the information system controls. Further, we reviewed the query documentation to ensure that all records and requested fields were included in the received data. We also checked the list for missing data, blank fields, and dates outside the audit period. In addition, we judgmentally selected a sample of 20 defect verifications from the list and compared the unique identification number, status, priority, problem, description, date, rail, line, and track segment to the original documents (subway track inspection reports). Further, we judgmentally selected a sample of 20 defect verifications from the subway track inspection reports and traced the unique identification number, enterprise asset management identification number, status, priority, problem, description, date, rail, line, and track segment to the list of defect verifications received from the MBTA’s defect module.
Based on the results of the data reliability procedures described above, we determined that the information obtained for the audit period was sufficiently reliable for the purposes of our audit.
| Date published: | March 10, 2025 |
|---|