The MBTA did not ensure that it documented inspections designed to evaluate Keolis’s performance in the areas of station maintenance, snow and ice removal, fare collection, and the availability of coaches and locomotives.
We noted the following issues with documentation:
- The MBTA does not maintain logs to document if and when it notifies Keolis to fix, repair, or clean a certain station or to remove snow and ice. No penalties were charged for station maintenance or snow and ice removal during the audit period.
- The MBTA only documents a fare collection inspection when an issue is identified. When an issue is identified, an “Operator Corrective Action Request” report is created and sent to Keolis. No “Operator Corrective Action Request” reports were created during the audit period. No penalties were charged for fare collection during the audit period.
- The MBTA does not document passed ADA inspections. Because of this, we were unable to test passed inspections to confirm that they were not failed inspections that the MBTA incorrectly marked as passed.
- As currently designed, the MBTA’s coach and locomotive availability reports within the Train Resource Management System (TRMS) do not document the time of day or the location of the asset. Without these key data points, the MBTA is unable to effectively monitor whether assets are available during peak service times at North and South divisions, as required by the contract. No penalties were charged for fleet availability during the audit period.
The MBTA and its customers are exposed to safety risks if the MBTA fails to properly manage resolution of issues at commuter rail stations and with regard to snow and ice. This can result in avoidable injuries or accidents and financial loss. Failure to properly document the management of ADA inspections can result in unnecessary inconvenience or denial of service to residents and can create avoidable financial risk for the MBTA. Failure to appropriately oversee the fare collection efforts of its contractor and coach and locomotive availability can result in the perception of unfairness for riders, reduction in service levels during peak times, and the loss of ridership and contractor-penalty revenue for the MBTA.
Proper documentation of inspections is crucial for evaluating performance and ensuring that agreed-upon service levels are met. Without these records, it can be impossible for the MBTA to properly manage its operations and challenging to address any potential shortcomings or hold the responsible parties accountable.
Authoritative Guidance
The MBTA is responsible for monitoring Keolis’s performance, according to Section 2 of Schedule 6.1 of the operating agreement. The operating agreement states,
The MBTA shall determine the level of the Operator’s performance in each Performance Area, . . . [within the Passenger Comfort and Service Amenities states,] periodic inspections conducted by the MBTA during the Reporting Period. The MBTA shall evaluate the Operator’s actual performance, as detailed in such Operator reports and MBTA inspection reports, and shall calculate the Performance Failure Payment.
Reasons for Issue
The MBTA does not have a policy or procedure in place that contains language regarding monitoring inspections being performed for station maintenance, snow and ice removal, ADA inspections, locomotive and coach availability, and fare collection.
Recommendations
- The MBTA should establish policies and procedures to monitor station maintenance, snow and ice removal, ADA inspections, locomotive and coach availability, and fare collections.
- The MBTA should document all inspections, regardless of outcome.
Auditee’s Response
The MBTA agrees that improvements can and are in the process of being made to the documentation process for certain types of inspections. That said the MBTA’s primary focus is, and will always be, ensuring safe, reliable, and accessible service to the MBTA’s customers and employees. The Draft Report states throughout the document that the SAO experienced constraints regarding its ability to obtain the information necessary to draw conclusions, specifically with regards to station maintenance, snow and ice removal, fare collection, fleet availability, and ADA inspections. The MBTA disagrees that the SAO’s audit was constrained. This audit has taken place over 14 months and the MBTA has provided responses to hundreds of requests for documents and information and the SAO has conducted dozens of interviews with MBTA and Keolis employees. The SAO’s finding itself is inconsistent with the idea that the audit was constrained. Again, the MBTA agrees it can improve its documentation of certain inspections, but this finding demonstrates that the SAO had sufficient information to conduct its audit and that it obtained sufficient evidence to provide a reasonable basis for its findings and conclusions.
Auditor’s Reply
We disagree with the MBTA’s claim that our audit was not constrained. For purposes of an audit, a “constraint” is a limitation on an auditor’s ability to answer the questions they set out to answer. This can occur for several reasons, including an auditee’s refusal to participate in the process or a lack of data provided. For the benefit of the reader, we will summarize the constraints we experienced below.
For our first objective, we sought to determine whether the MBTA ensured that Keolis met the performance requirements outlined in its contract. Specifically, we examined Keolis’s performance across several key areas, including on-time performance, seat availability, fleet availability, fleet maintenance, ADA compliance, and passenger comfort/service amenities. The latter encompasses aspects such as vehicle cleanliness, in-service amenities, train staffing, station maintenance, snow and ice removal, and fare collection.
As we explained in the "Audit Objectives, Scope, and Methodology” section of this audit report, and again in this finding, during our audit, we faced limitations in which essential information was either unavailable or not properly maintained by the MBTA. As a result, we were forced to limit the scope of our audit in certain areas, which impacted our ability to fully assess Keolis’s performance in those areas. These areas with limitations are outlined again below for the reader’s understanding.
The MBTA monitors Keolis’s ADA compliance by conducting physical inspections. It is authorized under its contract to penalize Keolis based on the number of failed ADA inspections. During our review, we requested data from MBTA management on all ADA inspections performed on Keolis during the audit period. However, MBTA management informed us that they do not retain records of inspections for which Keolis was deemed to have passed, only those that resulted in failure. As a result, we were unable to determine the total number of inspections performed or to assess whether any inspections that were deemed passed should have been failed, and whether any penalties should have been assessed accordingly. This resulted in a constraint in our audit.
Within its contract with Keolis, the MBTA has specified that a minimum number of locomotive and coach trains must be available for revenue service each day during the two daily weekday peak periods, which are from 5:00 a.m. to 10:00 a.m. and from 4:00 p.m. to 9:00 p.m. The northern and southern commuter rail divisions each have their own specific requirements for fleet availability. Both divisions must meet these requirements during the specified peak periods each day. If the required number of locomotives or coaches is not available for service on any given day, then Keolis should be assessed a penalty. According to MBTA officials, fleet availability is monitored through reports that Keolis is required to produce. However, when we requested these reports, we found that they did not capture key details, such as the vehicle location (north or south station) or the time of day that the vehicles were in use or available. As a result, we were unable to determine whether the vehicles were available during the peak periods and at the required locations each day, as specified in the contract. Instead, we were only able to determine whether the total number of vehicles met the required availability. This created a constraint on our ability to answer our audit questions.
Keolis is responsible for keeping all MBTA commuter rail stations free of trash, debris, and graffiti. The MBTA conducts weekly physical inspections of stations to ensure that Keolis meets this requirement. If any issues are found during an inspection, the MBTA will either open a service request or contact Keolis officials directly, prompting them to address the identified issues. Similarly, the MBTA monitors snow and ice removal through physical inspections when winter weather is expected to impact commuter rail service. If issues are identified during these inspections, then the MBTA contacts Keolis, prompting it to address the snow and ice removal concerns. However, the MBTA does not keep logs to document if and when it notifies Keolis to fix, repair, or clean a station, or to remove snow and ice. As a result, we were unable to determine how often such notifications occurred during the audit period or the extent to which they were addressed, constraining our ability to answer these questions.
The MBTA’s response highlights the 14-month duration of the audit, the hundreds of document requests, and the dozens of interviews conducted with MBTA and Keolis staff members. This is not directly relevant to the core issue, which is that limitations of available information imposed constraints on the audit. We recognize that the audit took longer to complete than originally expected and appreciate the MBTA’s participation with it. The extended time period required to conduct the audit was driven by several factors that are described in the “Other Matters” section of this report, including significant delays in the MBTA identifying the correct officials responsible for the programs under review, prolonged waits for requested information, and instances of inaccurate data provided by the MBTA.
| Date published: | March 4, 2025 |
|---|