Massachusetts Bay Transportation Authority - Keolis Contract - Finding 1

The Massachusetts Bay Transportation Authority did not accurately assess performance-based incentives and penalties for Keolis.

The Massachusetts Bay Transportation Authority (MBTA) did not accurately assess performance-based incentives and penalties for Keolis. Specifically, the MBTA failed to assess approximately $3,324,126 worth of performance-based penalties for Keolis, inappropriately assessed approximately $257,724 in performance-based penalties, overpaid Keolis by approximately $105,810 in performance-based incentives and underpaid Keolis by $105,210 in performance-based incentives.

Performance-based penalties and incentives are designed to incentivize vendors to achieve certain quality or timeliness standards and to hold them accountable for meeting agreed-upon service levels. Failing to accurately assess these penalties and incentives undermines accountability and reduces the incentive for vendors to meet expectations. It also diminishes the overall quality and reliability of the MBTA’s commuter rail services. In this instance, the MBTA failed to assess millions of dollars of financial penalties that could have helped improve service for MBTA customers. This represents a financial loss to the MBTA and could lead to other financial losses, as poorer service may result in fewer riders. Failure to properly assess incentives and penalties could also reduce the public’s trust in the MBTA and harm its relationship with a vendor that relied on the MBTA’s calculations of incentives and penalties. 

a. The Massachusetts Bay Transportation Authority did not assess approximately $3,324,126 worth of performance-based penalties for Keolis.

During the audit period, we found that the MBTA failed to assess Keolis $3,324,126 worth of performance-based penalties. These unenforced penalties spanned several different performance areas, as described in more detail below.

• On-time performance—we identified 17 commuter rail train trips that qualified for on-time performance penalties, which the MBTA did not assess. We also identified at least $54,829 in unassessed penalties related to on-time performance.
• Passenger comfort and service amenities—we identified 20 failed inspections for which the MBTA did not assess penalties for Keolis. These inspections totaled $20,499 in unassessed penalties. In addition, we identified 30 failed inspections where the MBTA under-penalized Keolis by a total of $147. In total, we identified at least $20,646 in unassessed penalties related to passenger comfort and service amenities.
• Fleet maintenance (locomotives)—we noted that, in 34 months out of 43 (79%), despite the MBTA conducting the minimum number of required fleet maintenance inspections necessary to assess penalties on Keolis, no penalties were assessed for locomotive maintenance. Within those 34 months, we identified 159 failed inspections that warranted penalties for Keolis, totaling $3,248,651 in unassessed penalties.

b. The Massachusetts Bay Transportation Authority inappropriately assessed approximately $257,724 worth of performance-based penalties against Keolis.

• Americans with Disabilities Act (ADA) compliance—we identified 70 failed inspections for which the MBTA inappropriately assessed penalties for Keolis in the amount of $227.
• Seat availability—we identified 572 train trips that did not meet the required number of seats. Our review of these trips identified $37,120 in inappropriately assessed penalties.
• Train staffing—we identified 1,604 train trips that did not meet the required number of train staff members. Our review of these understaffed train trips identified $25,346 in inappropriately assessed penalties.
• Fleet maintenance—

• Cab control coaches (CTC)—despite failing to conduct the minimum number of required fleet maintenance inspections necessary to assess penalties against Keolis in all 43 months during the audit period, we found that the MBTA inappropriately assessed $128,495 in penalties in this area.

• Blind trailer coach (BTC)—despite failing to conduct the minimum number of required fleet maintenance inspections necessary to assess penalties against Keolis in all 43 months during the audit period, we found that the MBTA inappropriately assessed $66,536 in penalties in this area.

c. The Massachusetts Bay Transportation Authority overpaid Keolis by approximately $105,810 in performance-based incentives.

• On-time performance—we found that the MBTA overpaid Keolis by $51 in on-time performance incentives. Specifically, we noted 8 trips that did not qualify for incentive payments.
• Seat availability—we found that the MBTA overpaid Keolis $105,675 in seat availability incentives. Specifically, we noted the following:

• a $105,210 overpayment due to an inadvertent clerical error, where the train seating and train staffing incentive amounts on a monthly invoice were mistakenly transposed and went uncorrected;

• overpayments worth $438 caused by MBTA invoicing errors that had gone undetected and were paid; and

• a $27 overpayment due to the incorrect escalation factor being used for fiscal year 2024.

• Train staffing—we found $84 worth of overpayments due to the MBTA inadvertently using the wrong escalation factor when calculating Keolis performance incentives in this area over a span of six months. We also identified 343 trips that did not qualify for incentive payments. These trips did not impact the incentive payment, as both the monthly and annual caps had already been reached. However, they were initially listed for payment and would have been paid out if the caps had not already been met.

d. The Massachusetts Bay Transportation Authority underpaid Keolis by $105,210 in performance-based incentives it had achieved.

We found that the MBTA underpaid Keolis by $105,210 in train staffing incentives. This underpayment resulted from the previously mentioned clerical error, where the incentive amounts for train seating and train staffing were mistakenly transposed on a monthly invoice and went uncorrected. This highlights a weakness in internal control and management oversight in this area. 

Authoritative Guidance

Section 13.1 of Part 1 of the MBTA contract with Keolis, dated February 5, 2014, states,

The MBTA may for its own purposes monitor or review proposals, plans or reports (or any aspect thereof) of the Operator under this Agreement, but no review, comment, statement, report or undertaking made or given by or on behalf of the MBTA during such review or monitoring (and no failure to undertake, make or give any review, comment or statement) shall operate to exclude or relieve either Party from or reduce or otherwise affect the obligations of such Party under this Agreement.

Sections 1.1, 1.2, 1.3, and 1.4 of Schedule 6.1 of the MBTA contract with Keolis, dated February 5, 2014, state,

The MBTA will evaluate the Operator’s On-Time Performance. . . . The MBTA will evaluate the Operator’s compliance with the consist requirements established by the MBTA for the North Division and the South Division Service Lines. . . . The MBTA will evaluate the Operator’s compliance with the standards established for maintenance of the revenue Rolling Stock Fleet based upon compliance with the state-of-good repair standards for locomotives, control coaches (“CTCs”) and trailer coaches (“BTCs”) following the prescribed 92-day maintenance interval for locomotives and CTCs, and the prescribed 184-day maintenance interval for BTCs (or the equivalent, as the same may be adjusted from time to time to conform to Applicable Law). . . . The MBTA will evaluate the Operator’s performance with respect to passenger comfort and service amenities.

Reasons for Issues

The MBTA did not have sufficient policies and procedures, including a monitoring component, to ensure that the performance-based incentives and penalties it assessed Keolis were accurate.

For on-time performance, we found that the MBTA applied an escalation method for on-time performance penalties that differed from the standard escalation used for all other performance area penalties, despite the contract not calling for a different method. In addition, TRMS data was not being updated accurately. For example, we identified instances in which trips recorded as late—which was done correctly—did not have any penalty applied in TRMS. Additionally, there were cases in which trips recorded as on-time in TRMS were actually not on time when we recalculated the arrival times (actual arrival time minus scheduled arrival time).

For passenger comfort and service amenities, we found that in some cases, the MBTA applied an incorrect penalty amount due to errors in calculating its penalty escalations. In one instance, we found that the MBTA allocated an incorrect number of points for one area on the inspection form, which resulted in the form being marked as passed when it should have been marked as failed.

For fleet maintenance (locomotives), there were instances in which the MBTA documented inspections as failed when they should have been passed. In addition, MBTA officials indicated that they would waive penalties for failed inspections caused by paperwork issues if Keolis presented the corrected paperwork to the MBTA. However, the MBTA inspection log was not updated with this information when penalties were waived.

For train seating penalties, in some cases, the MBTA applied an incorrect penalty amount due to errors in calculating the penalty escalations. In addition, the MBTA pulled TRMS reports before the close of the month in error, which led to penalties being applied based on incorrect data.

For train staffing penalties, in most cases, the MBTA applied an incorrect penalty amount due to errors in calculating the penalty escalations.

For fleet maintenance (coaches), MBTA officials explained in their response to this finding that they did not always record all inspections in Form.com and allowed inspections documented outside of Form.com to count toward meeting their minimum number of inspections. This practice makes it appear that they failed to meet the minimum number of inspections, which suggests that penalties should not have been assessed.

Recommendations

1. The MBTA should develop and implement sufficient policies and procedures, including a monitoring component, to ensure that the performance-based incentives and penalties it assesses Keolis are accurate and in accordance with the executed contract.
2. The MBTA should implement a verification process to ensure the correct escalation factors are used when calculating incentive and penalty amounts. This will help prevent errors and ensure accuracy in future calculations.
3. To help increase transparency and accountability, the MBTA should consider publicly reporting on the performance of its contractors. This would provide riders, taxpayers, and other stakeholders with insight into how contractors are performing relative to the MBTA’s expectations.

Auditee’s Response

The MBTA appreciates the work the [Office of the State Auditor (SAO)] has done in completing the Draft Report and as illustrated in more detail in our responses, the MBTA has already begun to make improvements based on some of the SAO’s recommendations and will continue to utilize the Draft Report and the final version in making improvements to the operations of the MBTA.

The MBTA received the engagement letter for this audit on September 11, 2023. The original audit period was from July 1, 2021 through June 30, 2023. On September 16, 2024 the audit period was expanded by the SAO to cover June 1, 2020 through December 31, 2023, with the reasoning provided that the extension “allows the audit period to include the amendment and extension to the MBTA’s ‘Commuter Rail Operating Agreement 159-12’ with Keolis, effective July 1, 2020. [SAO] extended the audit period through December 31, 2023, to capture more recent activity as of the time of our fieldwork.”¹

In 2014, the MBTA entered into an operating agreement with Keolis with an initial 8-year term ending June 30, 2022, plus extension options. In 2020, the MBTA exercised options to extend the term by four years, to June 30, 2026. Then in 2024, while preparing for the procurement of a successor contract, the MBTA extended the term by another year, to June 30, 2027 (for the purposes of this response, the Commuter Rail operating agreements and extensions will be referred to the “Operating Agreement”). Under the current Operating Agreement, the MBTA provides an annual fixed payment to [Keolis] for the performance of defined operations and maintenance functions; sets all fares and service schedules; conducts service planning and makes capital improvement decisions.

The MBTA Railroad Operations staff manages all aspects of service provision under the Operating Agreement through administration of the terms of the contract, monitoring of service and maintenance functions, enforcement of performance penalty and incentive regimes, regular meetings with the operator, and reporting through MBTA and contractor-managed information systems. As the MBTA’s commuter rail operator, [Keolis] employs and directs the commuter rail workforce of approximately 2,300 personnel; performs daily transportation operations (train crews, dispatching, etc.); maintains all MBTA-owned commuter rail assets (stations, track, signals, right of way, vehicle fleet, facilities, systems and tools); performs capital improvements compensated as supplemental work above the base operations and maintenance fee; performs all frontline customer service functions, including ticket sales, customer communications, disruption and special event management; and provides extensive reports and information about system operations to the MBTA. . . .

Thank you again for your consideration of the MBTA’s responses. As explained in more detail below, the MBTA has already started to implement some of the SAO’s recommendations. We look forward to working with the SAO on how the MBTA can continue to improve to provide safe, reliable, and accessible service to the MBTA’s customers and employees. . . .

The MBTA disagrees with some of the SAO’s calculations and interpretations of contract language and has provided a detailed breakdown in its responses to the exceptions provided by the SAO. . . .

For the on-time performance penalty amount . . . the MBTA in collaboration with Keolis automated the calculation of the on-time performance penalty escalation, using a multiplier approach rather than escalating each penalty fee individually. This accounts for the majority of the unassessed penalties. . . .

For the passenger comfort and service amenities inspections, the SAO identified 20 failed inspections in which penalties were not assessed and 30 failed inspections in which the MBTA under-penalized Keolis, totaling $20,646 in unassessed penalties. Of these 20 inspections identified, only 12 were failures and 8 were incorrectly written up as failed inspections, resulting in $7,573 in unassessed penalties. Further, of the 30 penalties where the incorrect penalty was used, one trip was classified as Class A rather than Class C cleaning, adjusting the alleged underassessment of $147 to an overassessment of $501.

For fleet maintenance, the SAO found 159 failed inspections totaling $3,248,651 in unassessed penalties. As previously reported to the SAO, the MBTA reviews the penalties with Keolis monthly, and adjustments are made based on the information provided at the time of the review. The SAO’s use and reference to “159 failed inspections” fails to consider this monthly review between the parties and clarifications discussed during those meetings. The lack of penalties assessed in connection with the 159 inspections referenced is not an example of an oversight or mistake on the part of MBTA, but rather a function of the outcome of the review process between the parties. In these instances, where no penalty was assessed, it was adjudged during those meetings there was no basis to do so, and the MBTA exercised its administrative discretion accordingly. The MBTA has provided documentation of the outcomes of the review process. The net variance for the penalty assessment for fleet maintenance is $0. . . .

For the Americans with Disabilities Act compliance the SAO identified 70 failed inspections for which the MBTA inappropriately assessed penalties to Keolis in the amount of $227. The MBTA agrees with the SAO’s analysis. . . .

For train staffing, the SAO identified 1,604 train trips that did not meet the required number of train staff members and identified $25,346 in inappropriately assessed penalties. The MBTA agrees that there was an overassessment of this amount, representing 1.8% of the total train staffing penalties paid of $1,396,889. . . .

For fleet maintenance, the SAO found that the MBTA failed to conduct the minimum number of required fleet maintenance inspections to assess penalties for both Cab Control Coaches (“CRC”) and Blind Trailer Coaches (“BTC”) and yet inappropriately assessed penalties in the amounts of $128,495 and $$66,536 respectively. The MBTA provided evidence to the SAO to demonstrate the minimum number of required fleet maintenance inspections were performed. For CRC, penalties were incorrectly assessed in October 2022, totaling $16,914 in overassessment and the MBTA used the incorrect escalation rate resulting in underassessment of $2,655. The net overassessment of penalties for CTC Inspections is $14,259. For BTC, penalties were incorrectly assessed in March 2021, October 2022, and July 2023, totaling $22,833 in overassessment and the MBTA used the incorrect escalation rate resulting in an underassessment of $1,041. The net overassessment of penalties for BTC Inspections is $23,874. The net overassessment of penalties for fleet inspections is $38,133. . . .

For on-time performance, the SAO found MBTA overpaid Keolis by $51 in incentive payments, with 30 trips that did not qualify for incentive payments. . . . The MBTA agrees with the net overpayment of incentives for on-time performance of $51.

For seat availability, the SAO found that 110 trips did not qualify for incentive payments, payments were made when the monthly incentive cap was met, Keolis’ invoicing errors went undetected and Keolis was overpaid $105,563 in overpaid seat availability incentives. The 110 trips did not affect the total incentive payments, as the monthly and annual caps had been achieved. As discussed in detail below, there was a clerical error in the transposition of the incentive amounts for seats and staffing in July 2022 leading to overassessment of $105,210 for seat availability. The offsetting value of negative $105,210 is reflected in subpart D. . . .

For train staffing, the SAO found the MBTA underpaid Keolis in train staffing incentives, specifically that the MBTA inadvertently used the wrong escalation factor, an invoicing error gone undetected by the MBTA, and that 343 trips did not qualify for incentive payments, totaling $105,126. . . . As discussed in the MBTA’s response to Finding 3, there was a clerical error in the transposition of the incentive amounts for seats and staffing in July 2022 leading to overassessment of $105,210 for seat availability.

[. . .]

1.   The SAO extended the audit period again with respect to its review of Keolis’s Disadvantaged Business Enterprise participation to cover July 1, 2014 through December 31, 2023.

Auditor’s Reply

For the passenger comfort and service amenities inspections, the MBTA suggests that eight cleaning inspections were performed after the coach’s entry to service, or midday. According to the MBTA, since midday cleanings are not subject to penalties, this misclassification led to the incorrect assessment of failures and $7,573 in unassessed penalties. We attempted to verify that these were, in fact, midday cleanings, but the MBTA could not provide sufficient evidence of when the inspection occurred. As such, we are unable to verify the accuracy of the MBTA’s statements in this regard.

In its response, the MBTA argues that it was justified in not imposing penalties for fleet maintenance, as adjustments were made during monthly meetings with Keolis and both parties agreed that penalties were not necessary. It claims to have provided documentation of this review process. However, the documentation, which was not provided until after our fieldwork ended, was insufficient. It shows that adjustments were made but does not explain why the decision was made not to impose penalties. This lack of clarity and transparency in the documentation made it impossible for us to assess whether these decisions were appropriate, and the failure to provide this information in a timely manner was an unfortunate pattern in this audit that occurred with troubling regularity.

The MBTA disagrees with our calculations of inappropriately assessed penalties related to coach maintenance, claiming that it has provided evidence showing that the required inspections were performed. The new information it provided to us contradicts previous statements made during the audit, upon which the MBTA told us to rely. Initially, MBTA officials explained that only inspections recorded in Form.com were considered for penalties. Now, after our audit has been completed, they are presenting an Excel document as additional evidence, despite having told us explicitly that it was not in use during the audit period. We have not had the opportunity to verify the veracity of the document, when it was developed, and other important factors. Since this documentation was provided after our fieldwork and we did not have the opportunity to assess its reliability, we are unable to consider it as sufficient evidence and must reject it.

Our audit began on November 9, 2023. We conducted an exit conference with MBTA management on November 1, 2024 to present our preliminary findings and recommendations. The purpose of this meeting was to facilitate a discussion and clarify any potential issues, providing MBTA officials an opportunity to express their views on the audit results. MBTA officials did not have any questions or comments regarding this finding during this meeting. It was not until six weeks later, on December 18, 2024, that the MBTA began providing us with additional information that it requested us to consider regarding this finding. We submitted our draft report to the MBTA on December 27, 2024, and allowed 15 days for its written responses. On January 2, 2025, MBTA officials provided us with more new information, which it did not previously provide to us during our audit. We had requested this information many months before but were only provided with it two months after completing our audit and after presenting our conclusions to the MBTA. This was followed by additional information on January 10 and 13. On January 15, we received the MBTA’s formal, written responses to the draft report. The MBTA continued to provide us with additional information, on a rolling basis, until January 31, 2025. After reviewing all the information provided, including the written responses, we agreed, as a courtesy, to consider it and assess its validity. As a result, adjustments were made to this finding. On February 5, the MBTA sent us another email containing further additional information related to this finding. This last submission could not be incorporated because it would have required additional validation and potential revisions, which would have further extended our review process and further delayed the issuance of this report. Per generally accepted government auditing standards, if an auditee does not respond within a “reasonable period of time” (in this case, the 15 days required by Section 12 of Chapter 11 of the Massachusetts General Laws), then auditors may issue the report without the auditee’s comments. As listed in the above chronology, the MBTA was allowed much more than a “reasonable period of time” to provide any pertinent information related to this finding.

We provided the MBTA with ample time to provide the requested information in response to our audit and the questions we asked. As discussed further in this report, the MBTA staff members assigned to our audit had little or no knowledge, experience, or expertise in the areas we were examining. After later changing its team, the MBTA’s new staff members who were assigned to our audit provided us with information that the MBTA now says is inaccurate and unreliable, even though the MBTA initially stated that we could and should rely on it for the purposes of our audit. We understand that audits can be difficult and time-consuming, but they are critical to transparency and public support of government programs. The fact that the MBTA was unable to provide us with accurate information means that the MBTA itself may not know what information is accurate—that is, it is unable to properly manage this contract and its $2,686,344,294 in public spending—or that the MBTA intentionally provided us with erroneous information. We find either scenario highly troubling.

According to its response, the MBTA has already begun to make improvements based on some of our recommendations and will continue to use this report in making improvements to the operations of the MBTA. We will review progress on this issue in our post-audit review in six months, when we hope to receive significantly improved cooperation from the MBTA.

Date published:	March 4, 2025