Audit of the Massachusetts Bay Transportation Authority's Contract Management Process Scope, Objectives and Methodology

An overview of the purpose and process of auditing of the Massachusetts Bay Transportation Authority's contract management process

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Massachusetts Bay Transportation Authority (MBTA) for the period July 1, 2014 through December 31, 2016.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

The scope of our audit was to determine whether the MBTA had taken measures to adequately address the management action items identified by an external auditor. These action items were in the areas of the MBTA’s bid and award, change order,² and cost estimation processes.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Has the MBTA performed market research to increase the number of contractors to bid on proposed MBTA work?	Yes
Has the research on disinterested bidders been captured and recorded accurately in the Disinterested Bidders Database?	No; see Finding 2
Does the MBTA approve and sign off on the statement of compliance for all change order costs?	Yes
Does the MBTA ensure that each change order adheres to the terms and conditions of the contract?	Yes
Does the MBTA administer the change order process in compliance with its Construction Contract Change Order Guidelines?	No; see Finding 3
Does the MBTA use the Association for the Advancement of Cost Engineering (AACE) methodology for determining estimating accuracy?	Yes
Has the MBTA adhered to its project controls regarding estimates and risk analysis?	No; see Finding 1
Does the MBTA maintain an accurate and reliable Bid Comparison to Cost Estimate Database?3	No; see Finding 4a
Does the MBTA perform a root cause analysis4 for each project bid that varies from the engineer’s estimate5 by more than 10%?	No; see Finding 4b

To achieve our audit objectives, we gained an understanding of the internal controls we determined to be relevant to our audit objectives and tested the operating effectiveness of those controls over the bid and award process, the change order process, and the cost estimation process. We conducted further audit testing as described in the following subsections.

Bid and Award—Disinterested Bidders Database

There were 39 construction contracts awarded during the audit period, which had 170 prequalified bidders (contractors that have been evaluated by the MBTA and found to be qualified to bid on MBTA projects). Of those 170, 139 submitted bids, according to the MBTA Bidders Database. The other 31 were “disinterested bidders,” which the MBTA defines as bidders who were prequalified and submitted proposals but elected not to submit bids. We determined whether information about the 31 disinterested bidders was entered in the MBTA’s Disinterested Bidders Database.

• We examined the Disinterested Bidders Database to determine whether market research was performed during the audit period.
• Twenty-nine of the 39 awarded construction contracts were design-bid-build (DBB)⁶ projects. We inspected the Planholders⁷ Lists for the 29 DBB projects and identified 883 Other Planholders that the MBTA did not review to identify potential new prequalified bidders.
• For the 34 of the 39 construction contracts that were competitively awarded during the audit period,⁸ we determined the number of bids submitted for each contract to determine whether the average number of bids received increased or decreased during the audit period.
• We identified the winning bidders on the 39 contracts awarded to determine how many different contractors received contract awards during the audit period.

Change Orders

• From a total population of 356 change orders executed during the audit period, we tested a nonstatistical, random sample of 40, with a total value of $4.3 million, to obtain reasonable assurance that proper approval and signatures were obtained. We inspected the sample to determine whether follow-up reviews and signoffs of MBTA Construction Form 3C⁹ were performed in accordance with Section 39I of Chapter 30 of the General Laws and within the authorization limitations of contract authority provided by the regulations of the MBTA board of directors before payment.
• From a total population of 356 change orders executed during the audit period, we tested a nonstatistical, random sample of 40, with a total value of $4.3 million, to determine whether the final change order packages contained the supporting documentation needed in order to complete the change order process, including the contractor’s proposal cover letter, recapitulation chart, Record of Negotiation, Independent Cost Estimate, Explanation of Necessity, and Contractor’s Change Order / Risk Reallocation Check-Off List.
• We reviewed all the documentation for these 40 change orders and calculated the dollar value of change order costs paid without adequate supporting documentation.

Project Estimating and Risk Analysis

Of the 29 DBB projects the MBTA awarded during our audit period, we tested a nonstatistical, random sample of 10 to determine whether the MBTA or its contracted design engineer prepared required project cost estimates and risk analyses as required by MBTA guidelines and whether all the documentation the AACE methodology requires was completed when appropriate.

We tested 10 projects to determine whether the following were prepared:

• Initial Project Estimates
• Pre-Conceptual Project Estimates
• 15% Conceptual Construction Estimates
• 30% Design Development Construction Estimates
• 60% Design Development Construction Estimates
• 100% Construction Documents Construction Estimates
• Cost Growth Reports¹⁰
• Bases of Estimates¹¹
• Qualitative Risk Analyses and Risk Registers¹² at the 30% Design Development Construction Estimate phase
• Quantitative Risk Analyses and Risk Registers between the 90% Design Development Construction Estimate phase and 100% Construction Documents Construction Estimate phase

Construction Contract Bid Analysis

• We tested the data for all 29 DBB projects recorded in the Bid Comparison to Cost Estimate Database to the verified Contract Bidder Info, Winning Bids¹³ Database for accuracy.
• We tested all 15 of the contracts that required a root cause analysis on an MBTA-prepared list that identified each bid on which a root cause analysis was performed during the audit period to obtain reasonable assurance that the MBTA’s contracted design engineers properly performed that analysis.

Whenever sampling was used, we applied a nonstatistical approach, and as a result, we were not able to project our results to the entire population.

Data Reliability

We obtained the following data sets from the MBTA’s Capital Management System database application, and we assessed the reliability of the data we obtained from the following databases: Disinterested Bidders, Bidders, Awarded Contracts, Change Orders, Engineer Estimate and Actual Cost Variance, Bid Comparison to Cost Estimates, and Contract Bidder Info All Bids. We reviewed the controls for access to programs and data, program changes, and security settings. We also performed additional validity and integrity tests, including testing the accuracy of manually entered data, by selecting a nonstatistical, random sample of source documentation and tracing applications to the data in the database. We determined that the data from the system were sufficiently reliable for the purposes of our audit.

2.    A change order is a document requesting, acknowledging, and approving (or disapproving) work that is either added to or deleted from the project scope, altering the original guaranteed maximum price in the contract, the planned completion date, or both.

3.    The MBTA maintains this database to compare engineers’ estimates to all project bids for all DBB projects and identify all bids varying from the engineer’s estimate by more than 10% for each DBB project.

4.    This analysis is MBTA’s process for determining the reasons for variances between the engineer’s estimate and bids received.

5.    This is the final construction cost estimate for a project and is prepared by a design engineer’s cost estimator.

6.    In a DBB project, the MBTA hires a contractor to create design specifications for a project and then uses those specifications to solicit bids for the construction of the project.

7.    A planholder is a person or company that requests a copy of an MBTA project plan.

8.    The other 5 contracts were not subject to competitive bidding requirements.

9.    This form is used to document a contractor’s proposed change to a contract and must be reviewed and approved by the MBTA project manager and, depending on the cost of the proposed change, various other MBTA personnel.

10.    This report is a table created by the design engineer that identifies and describes the factors that contributed to the growth of a project’s costs.

11.    A Basis of Estimate is a collection of documentation that is prepared by the design engineer and includes a project scope description, general assumed costs, pricing details, anticipated allowances, potential cost items excluded from the estimate, historical comparisons, areas of significant risk and opportunities, and a list of estimating team members.

12.    This is a document developed to identify schedule and cost risks associated with a project.

13.    This database identifies all winning construction contractors for all MBTA project types and includes data such as the contract number, project type, winning contractor, date of award, engineer’s estimate, award amount, and variance between the engineer’s estimate and the award amount.  

Date published:	September 28, 2018