Audit of the Massachusetts Bay Transportation Authority—Automated Fare Collection System 2.0 Project Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Massachusetts Bay Transportation Authority—Automated Fare Collection System 2.0 Project.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Massachusetts Bay Transportation Authority (MBTA) for the period January 1, 2018 through September 30, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Does the MBTA follow the submittal review period stated in Section 8 of Appendix 7 of the Amended and Restated Project Agreement to determine whether Boston [Automated Fare Collection System (AFC)] 2.0 OpCo LLC is in compliance?	Partially; see Finding 1
Does the MBTA have a security bond in place for the Amended and Restated Project Agreement and ensure that this bond is provided by an entity with no financial connection to the AFC 2.0 project contractor as good industry practices?	No; see Finding 2
To what extent is the MBTA using MBTA employees to assist with the completion of the AFC 2.0 project?	To some extent; see Findings 3 and 4

To achieve our objectives, we gained an understanding of the aspects of the MBTA’s internal control environment that we determined to be relevant to our objectives by reviewing applicable policies and procedures and relevant contracts, by conducting interviews with management, and by reviewing the processes related to our objectives. We evaluated the design of controls over the oversight of development and construction contracts to ensure that contractors obtained bonds and guarantees, and the administration of MBTA employees working on the AFC 2.0 project. We evaluated the design and tested the operating effectiveness of controls over the approval of submittals for development work on the AFC 2.0 project. We selected a random, statistical sample of 47 submittals from a population of 1,001 and determined whether an approved MBTA representative signed the submittal response letters. All of the letters in our sample were signed by approved MBTA representatives.

To obtain sufficient, appropriate evidence to address our audit objectives, we performed the audit procedures described below.

Submittals

To determine whether the MBTA followed the submittal review period stated in Section 8 of Appendix 7 of the Amended and Restated Project Agreement and to determine whether AFC 2.0 project contractors were in compliance, we took the following actions.

We requested, and the MBTA provided us with, submittal data for the AFC 2.0 project. The MBTA uses an Excel spreadsheet called the Submittal Tracking Tool that captures all work orders and change orders performed by the Systems Integrator (SI) to develop the AFC 2.0 system. We reviewed this data and found that, during the audit period, there were 1,001 distinct submittals, which included project milestone dates, SI status, MBTA review status, scheduled review dates, and actual delivery and review dates. We selected a random, statistical sample of 93 submittals from the population of 1,001 using a 95% confidence level,⁵ a 0% expected error rate,⁶ and a 5% tolerable error rate.⁷ We used TeamMate Analytics⁸ to select the sample. For our sample population, we determined whether the MBTA received submission letters from the SI for completed work. We also reviewed copies of the MBTA response letters the MBTA sent to the SI that indicated whether the MBTA accepted the work performed or sent it back for further work. We captured the dates of these submission and response letters to calculate the number of days it took the MBTA to respond to the SI to determine whether they fell within the compliance period.

We did not identify any exceptions in our testing; therefore, we concluded that, during the audit period, the MBTA followed the submittal review period stated in Section 8 of Appendix 7 of the Amended and Restated Project Agreement to determine that the AFC 2.0 project contractors were in compliance. However, we noted that the Amended and Restated Project Agreement contract allows deadlines to be overridden by the MBTA and Boston AFC 2.0 OpCo LLC. See Finding 1.

Bonds and Guarantees

To determine whether the MBTA ensured that Cubic and McDonald had obtained bonds or other security guarantees for the work they were performing on the AFC 2.0 project, we obtained a list of the 143 vendors that the MBTA used on the AFC 2.0 project. We could not determine which of the vendors worked specifically on the AFC 2.0 contract because the work performed by many of the vendors was for multiple projects besides the AFC 2.0 project. We selected to examine the contract with Cubic as it was the primary vendor of the AFC 2.0 contract and McDonald which, as construction vendors, would both have a bond in accordance with Section 29 of Chapter 149 of the General Laws. We requested the contracts from the MBTA for these two contractors and any bonds or other forms of surety that would benefit the MBTA if the contractors defaulted on the contract for work on the AFC 2.0 project. We reviewed the two contracts we received and determined whether the bonds or other surety given were in compliance with the Amended and Restated Project Agreement. We also obtained the names of the bonding or surety companies for these contractors. We identified the parent companies of the bonding or surety companies and reviewed their audited financial statements to determine whether there were any disclosures related to Cubic or McDonald. We determined that these companies were international publicly traded insurance companies and there were no related party transactions with the contractors.

We noted that the MBTA did not ensure that Boston AFC 2.0 OpCo LLC obtained a surety bond as required by Section 29 of Chapter 149 of the General Laws. See Finding 2. 

Using MBTA Employees to Perform Work on the AFC 2.0 Project

To determine whether the MBTA had assigned its own employees to work on the AFC 2.0 project, we interviewed MBTA employees about the AFC 2.0 project. We obtained employee data from the Human Resources Compensation Management System (HR/CMS), which captured 44 employees who charged time to the AFC 2.0 project during the audit period. This list was missing 10 employees who were reflected in the MBTA’s Financial Management Information System (FMIS) data. We summarized HR/CMS wages paid to employees during the audit period. We obtained employee wage data from the MBTA’s FMIS for 48 employees charging time to the AFC 2.0 project during the audit period. This list was missing 6 employees who were reflected in the HR/CMS data. We summarized FMIS wages paid to employees during the audit period. We compared the wages paid to the 54 employees between HR/CMS and FMIS and identified a variance of $349,523 (HR/CMS had a larger total than FMIS). We requested a reconciliation from the MBTA; however, the MBTA was unable to provide a reconciliation to us.

During our testing, we noted that the data for employees is not consistent between HR/CMS and FMIS. In fact, we identified a discrepancy of $349,523 between the two systems. See Finding 3.

Data Reliability Assessment

To determine the reliability of data on the MBTA spreadsheets of submittals during the audit period, we interviewed management who were responsible for the source data. Further, we used electronic spreadsheet functionality to identify hidden cells, rows, and/or errors, noting no exceptions. We selected a random sample of 20 submittals from the Submittal Tracking Tool spreadsheet and compared the information from the list to the actual source (Asana, Box, and other repository folders). Comparison criteria included information such as submittal identification number and submittal name/description. We also selected a random sample of 20 submittals from the source (Asana) and compared the aforementioned information to the Submittal Tracking Tool spreadsheet.

To determine the reliability of the data the MBTA provided to us for its vendors, we requested a query of the spending for vendors from FMIS. We were unable to determine the completeness and accuracy of the list of vendors because the work performed by these vendors was not always exclusively for the AFC 2.0 project. See Finding 2.

To determine the reliability of the employee data from HR/CMS and FMIS, we tested the data for duplicates and missing data. We also tested select system controls for both systems related to access controls, configuration management, contingency planning, segregation of duties, and security management. See Finding 4. We attempted to trace a sample of 10 MBTA employees to source documentation; however, the MBTA did not provide documentation regarding wage information. We used the lists of employees that we obtained directly from MBTA, as that was the only source available for our audit purposes. See Finding 3.

Based on the results of the data reliability assessment procedures described above, we determined that the information we obtained was sufficiently reliable for the purposes of our audit.

5.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

6.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

7.    Tolerable error rate is the maximum error in the population that auditors would be willing to accept and still conclude that the result from the sample has achieved the audit objective.

8.    This is a Microsoft Excel–based data analytics tool that allows auditors to execute advanced data analysis.

Date published:	January 16, 2025