Audit of UMass Memorial Health Care Inc. Objectives, Scope, and Methodology

An overview of the purpose and process of auditing UMass Memorial Health Care Inc.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor (OSA) has conducted a performance audit of certain activities of UMass Memorial Health Care (UMMH) for the period July 1, 2020 through June 30, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Did UMMH spend grant funds from the Executive Office of Health and Human Services (EOHHS) in accordance with Chapter 102 of the Acts of 2021 and its Payment Agreement with EOHHS?	No; see Finding 1
Did UMMH spend grant funds it received from the Massachusetts Emergency Management Agency (MEMA) in accordance with the associated grant applications?	Yes
Did UMMH provide inpatient maternity services to MassHealth members, specifically the following: documenting the provision of services payable to MassHealth in accordance with Section 450.205(A) of Title 130 of the Code of Massachusetts Regulations (CMR); having registered nurses present during and immediately after delivery in accordance with 105 CMR 130.616(F) and having neonatal nurses assigned to no more than three patients at one time in accordance with best practices established by the Association of Women’s Health, Obstetric and Neonatal Nurses; and reporting births that occurred at UMMH hospitals to the Department of Public Health (DPH) within 10 days of the birth in accordance with 105 CMR 305.020?	No; see Finding 2

To accomplish our objectives, we gained an understanding of the internal control environment relevant to our objectives by reviewing applicable policies and procedures and UMMH’s internal control plan, as well as by conducting inquiries with UMMH officials. In addition, to obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

EOHHS Grant Spending

We identified a total of two payments, totaling $6,200,807, that UMMH received from EOHHS and were authorized by Chapter 102 of the Acts of 2021.

We requested supporting documentation for both payments in order to determine whether these funds were used in accordance with UMMH’s Payment Agreement with EOHHS. UMMH, through legal counsel, stated that it did not maintain any documentation to support the expenditures from the $6.2 million received from EOHHS and stated that these funds were used to generally recover from costs associated with COVID-19. After our exit conference, which occurred five months after our request for this data, UMMH revised its response to state that the entire $6.2 million was used to reimburse itself for employee bonuses paid in 2021 and 2022 and provided us with an Excel spreadsheet that listed the bonuses paid to UMMH employees during that time. However, we could not verify the accuracy of these bonuses because UMMH did not provide us records to verify that the grant payments were used for this purpose.

See Finding 1 for more information.

MEMA Grant Spending

We identified a total of 15 grant payments, totaling $25,357,758, that UMMH received from MEMA and were authorized by the Federal Emergency Management Agency.

We reviewed grant applications and completed project forms for all 15 grant payments to determine whether the total projected costs from the applications matched the total amount spent from the grant. We also reviewed each completed project form to determine whether the amount spent was approved by a Federal Emergency Management Agency representative.

Based on the results of our testing, we determined that all grant payments received during the audit period were spent in accordance with the grant applications.

Birth Reporting to DPH

We selected a random, statistical² sample of 74 out of 4,989 births by MassHealth members that occurred in a UMMH hospital during the audit period. We used a 90% confidence level,³ a 50% expected error rate,⁴ and a 20% tolerable error rate.⁵

We reviewed birth logbooks and emails that the UMMH birth registrar sent to DPH to determine whether UMMH reported all 74 births in our sample to DPH within 10 days, as required by 105 CMR 305.020.

We also reviewed service notes for all 74 sampled births to determine whether the provision of services was documented in accordance with MassHealth regulations.

See Finding 2 for more information. 

Registered Nurse Staffing

We selected a random, statistical sample of 74 days out of a population of 1,089 days where a MassHealth member gave birth in a UMMH hospital during the audit period. There were 348 deliveries by MassHealth members that occurred during the sampled 74 days. We used a 90% confidence level, a 50% expected error rate, and a 20% tolerable error rate.

We requested a list of registered nurses who were assigned to each of the 348 deliveries by MassHealth members during the sampled 74 days to determine whether a registered nurse was present during and immediately after each delivery and to determine the number of patients who were MassHealth members that each registered nurse was assigned on all 74 sampled days.

We analyzed the data provided by UMMH to determine whether there were two registered nurses assigned to each of the 348 deliveries in our sample, in accordance with 105 CMR 130.616(f). We also analyzed the data provided to us by UMMH to determine whether the number of deliveries assigned to each registered nurse per day was consistent with best practices established by the Association of Women’s Health, Obstetric and Neonatal Nurses.

Data Reliability

We obtained data for EOHHS and MEMA grants from CTHRU, the platform used by the Office of the Comptroller of the Commonwealth to report state spending. CTHRU data is sourced directly from the Massachusetts Accounting and Reporting System (MMARS). To determine the reliability of this data, we relied on the work performed by OSA in a separate project in 2022. As part of that work, OSA reviewed existing information, tested selected system controls, and interviewed knowledgeable officials at the Office of the Comptroller of the Commonwealth about MMARS. As part of our current audit, we performed validity and integrity tests on the grant fund data, including (1) testing for duplicate records, (2) testing for blank fields, and (3) testing for dates outside the audit period. Next, for all EOHHS grant payments, we matched the amount, vendor name, and appropriation code from the Payment Agreement to the CTHRU data. Next, for all 15 MEMA grant payments, we matched the vendor name, vendor city, and payment amount from the grant applications to the MMARS data.

We obtained data for all deliveries by MassHealth patients at UMMH’s hospitals directly from UMMH, which queried the information using Epic Systems software. To determine the reliability of the data, we (1) interviewed UMMH’s vice president of quality informatics to gain an understanding of the software, (2) reviewed System and Organization Control-3 reports⁶ for Epic Systems, (3) reviewed policies and procedures related to system access and software use, and (4) tested information system access controls and security awareness training. Additionally, we randomly selected 20 births from the Epic Systems data and determined whether the information matched the source documentation from UMMH for each delivery. Next, we randomly selected a sample of 20 source documents and traced the information to the Epic Systems data.

Based on the results of the data reliability assessment procedures described above, we determined that the information we obtained, except for the information security awareness training issues noted in Other Matters, was sufficiently reliable for the purposes of our audit.

2.    Auditors use statistical sampling to select items for audit testing when a population is large (usually over 1,000) and contains similar items. Auditors generally use a statistics software program to choose a random sample when statistical sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

3.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

4.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

5.    The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

6.    A System and Organization Control report is a report on controls about a service organization’s systems relevant to security, availability, processing integrity, confidentiality, or privacy issued by an independent contractor.

Date published:	November 4, 2025