Overview
The Massachusetts Bay Transportation Authority (MBTA) did not ensure that its own staff, and its contracted design engineers and their estimators, properly conducted periodic cost estimates and risk analyses to evaluate potential threats to successful project completion throughout each capital construction project. The purposes of these cost estimates and risk analyses are to determine whether a project is progressing as planned and to ensure that any problems related to project costs or scheduled completion timeframes are identified and resolved promptly. Because it did not always conduct them, the agency did not collect all the information it needed to effectively and efficiently administer projects, which could result in problems such as project delays, cost overruns, and funding issues.
We selected a nonstatistical, random sample of 10 of the 29 design-bid-build (DBB) construction projects that the MBTA initiated during our audit period. The MBTA did not provide us with the following materials:
- Initial Project Estimates for 10 projects
- Pre-Conceptual Project Estimates for 10 projects
- 15% Conceptual Construction Estimates for 10 projects
- 30% Design Development Construction Estimates for 6 projects
- 60% Design Development Construction Estimates for 7 projects
- a 100% Construction Documents Construction Estimate for 1 project
- Cost Growth Reports at any design estimate phase for 10 projects
- a Basis of Estimate for 1 project
- Qualitative Risk Analyses at the 30% Design Development Construction Estimate phase for 10 projects, or Risk Registers for the same 10 projects
- Quantitative Risk Analyses between the 90% Design Development Construction Estimate phase and 100% Construction Documents Construction Estimate phase for 3 projects, or Risk Registers for the same 3 projects
Authoritative Guidance
According to the “Estimating Requirements” subsection of Section II of the MBTA Project Controls Manual, which was released October 16, 2014 and in effect during our audit period, estimates are required for MBTA construction projects at specific design phases:
- Initial Project Estimate: an initial estimate of what the total project budget will be, broken down into projected costs for construction, design, real estate, project administration, etc., that is developed by the MBTA project manager (PM)
- Pre-Conceptual Project Estimate: the PM’s revision of the Initial Project Estimate after a designer is selected, if necessary
- 15% Conceptual Construction Estimate: an updated version of the estimated costs, produced by the PM, based on concept schemes submitted by the design engineer and its estimator, when the project design is 15% complete
- 30%, 60%, and 90% Design Development Construction Estimate: construction estimates of the MBTA’s preferred design scheme, produced by the design engineer and its estimator, that account for all cost-sensitive project data, submitted when the project design is 30%, 60%, and 90% complete
- 100% Construction Documents Construction Estimate: a detailed construction project estimate that is produced by the design engineer and its estimator and submitted to MBTA management when the project design is 100% complete
The “Design and Construction Phase Cost Estimating” subsection of Section II of the manual states, “A Cost Growth Report (CGR) is to be provided to the MBTA with every construction estimate submittal in order to track cost growth at each design phase by comparison with the project budget.”
The “Requirements for Basis of Estimate” subsection of Section II states, “A Basis of Estimate shall accompany all estimates.”
The “Risk Analysis Process” subsection of Section IX of the manual states, “At 30% [design], a qualitative risk analysis is required. The output of a qualitative risk analysis is a risk register.”
The “Risk Analysis Process” subsection of Section IX of the manual states, “When the project design is between 90% and 100%, a quantitative risk analysis is required. The output of a quantitative risk analysis is . . . a risk register.”
Reasons for Noncompliance
MBTA management has not established effective monitoring controls to ensure that periodic project estimates and risk analyses are properly conducted during various design phases of MBTA capital construction projects as required by the Project Controls Manual.
Recommendation
MBTA management should establish effective monitoring controls to ensure that periodic project estimates and risk analyses are properly conducted by MBTA personnel, its contracted design engineers, and their estimators.
Auditee’s Response
In a response to this finding dated July 26, 2018, the MBTA stated,
To ensure that new and existing Department staff are knowledgeable in the requirements for project cost estimates and risk analyses, the Department has developed a new on-boarding procedure for new hires that includes a review of several different Capital Delivery functions and processes, including items identified in the Draft Audit finding. This information is not only available to new hires, but is also being rolled out to the entire Capital Delivery Project staff. . . .
The MBTA continues to improve its estimating and risk analyses procedures and documentation. For example, the Capital Investment Plan (CIP) process now includes using a standard Capital Funding Request template for new projects and requires initial estimates of the project budget and schedule. After the CIP is approved but before funding is authorized, project teams are also required to submit a Project Information Form with updated cost estimates and schedule projections. This new process requires submission and review of an initial project budget.
The MBTA is renewing its focus on making certain that interim and final estimates and risk analyses are performed according to the requirements of the MBTA Project Managers (PM) Manual and MBTA Project Controls Manual and are well documented. It should be noted that as a result of the audit process it was found that there are inconsistencies between the MBTA PM Manual and MBTA Project Controls Manual. In some instances, a requirement in one manual was not specified in the other. An example of this was the requirement for a Cost Growth Report that is specified in the Project Controls Manual but not the PM Manual.
The Capital Delivery Assistant General Manager has engaged the Capital Delivery Department Director of Quality Assurance and Quality Control to review both Manuals, identify inconsistencies and update each Manual accordingly. This analysis has begun. The Department’s [Quality Assurance / Quality Control] Group is leading a complete review of the manuals by engaging the stakeholders in the Capital Delivery and Capital Program Oversight Departments. The Department acknowledges that the Cost Growth Report, while not required in the current PM Manual, is an important tool to evaluate cost increases. Through the review of the PM and Project Controls manuals this process will be evaluated, modified, and required for all projects that show an increase in cost as the design progresses.
With its written response to our draft report, the MBTA also provided copies of the following documents:
- 8 of the 10 missing Initial Project Estimates
- 3 of the 10 missing 15% Conceptual Construction Estimates
- 2 of the 6 missing 30% Design Development Construction Estimates
- 3 of the 7 missing 60% Design Development Construction Estimates
- the missing Basis of Estimate
- 1 of the 3 missing Quantitative Risk Analyses between the 90% Design Development Construction Estimate phase and 100% Construction Documents Construction Estimate phase
- 2 of the 3 missing Risk Registers for the Quantitative Risk Analyses between the 90% Design Development Construction Estimate phase and 100% Construction Documents Construction Estimate phase
In addition to these documents, the MBTA also provided explanations of why some of the documentation was still missing. For example, regarding the missing Pre-Conceptual Project Estimates, the MBTA stated in its July 26, 2018 response to the draft report,
This is a requirement of the Project Controls Manual but not the Project Managers Manual. The MBTA is doing a review of both manuals for consistency and will make a determination as to need and will revise both manuals accordingly.
Regarding Cost Growth Reports, the MBTA stated,
The development of a Cost Growth Report is required as indicated on Page 14 of the Project Controls Manual, but was not identified in the Project Managers Manual. This inconsistency will be addressed in the review of the two manuals.
To explain the missing 15% Conceptual Construction Estimates, 30% Design Development Construction Estimates, and 60% Design Development Construction Estimates, the MBTA stated that to expedite a project, depending on the project scope, these estimates might not be performed. It also stated,
In many instances the 30% design submittal may not have been required. Regardless, early identification of project schedule and cost risks are a good practice and will be completed and documented in future 30% design submissions.
Auditor’s Reply
As noted above, during our audit, the MBTA could not provide various documents to substantiate that its own staff, as well as its contracted design engineers and their estimators, properly conducted periodic cost estimates and risk analyses. The MBTA did eventually provide additional documents as well as explanations of why certain documents were missing. However, since this documentation and related information were not provided to the Office of the State Auditor (OSA) during our audit fieldwork, we could not perform the testing necessary to verify their accuracy and authenticity. Even if all the records the MBTA provided with its response are accurate, there is still a significant number of instances where there is no documentation to substantiate that required periodic cost estimates and risk analyses were conducted. Although the MBTA acknowledges that, in a number of instances, estimates were forgone in order to expedite projects, OSA believes these estimates are integral to the effective administration of these projects, are required by MBTA policy for all construction projects, and therefore should have been performed.
Based on its response, the MBTA is taking measures to address our concerns in this area. However, we again recommend that the MBTA establish effective monitoring controls to ensure that periodic project estimates and risk analyses are properly conducted and documented during various design phases of its capital construction projects as required by the MBTA Project Controls Manual.
| Date published: | September 28, 2018 |
|---|