Audit of the Massachusetts Emergency Management Agency Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Massachusetts Emergency Management Agency.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Massachusetts Emergency Management Agency (MEMA) for the period March 1, 2020 through June 30, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective was discussed in the audit findings.

Objective	Conclusion
Did MEMA fulfill COVID-19 resource requests accurately and timely in accordance with Sections 3.1 and 3.2 of MEMA’s “Concept of Operations for Managing Resource Requests”?	Yes
Did MEMA assist applicants with public assistance grant applications related to COVID-19 in accordance with certain aspects of Section IV(A) numbers 1, 8, 12, 13, 14, 17, and 21 of MEMA’s “State Administrative Plan for the [Federal Emergency Management Agency (FEMA)] Public Assistance Grant Program”?	Yes
To what extent did MEMA collaborate with the Operational Services Division (OSD) and the Department of Public Health (DPH) to coordinate COVID-19 resource management and delivery?	To a sufficient extent
Did MEMA have controls in place over the receipt, expense, and reporting of COVID-19 funds in accordance with MEMA’s reimbursement and payment practices?	No; see Finding 1

To achieve our audit objectives, we gained an understanding of the internal control environment related to the objectives by reviewing applicable MEMA policies and procedures, MEMA’s internal control plan, conducting interviews with MEMA management and employees, and performing walkthroughs of the processes related to the administration of COVID-19 funds.

To obtain sufficient, appropriate evidence to address our audit objectives, we performed the audit procedures described below.

COVID-19 Resource Requests

To determine whether MEMA fulfilled COVID-19 resource requests accurately and timely in accordance with Sections 3.1 and 3.2 of MEMA’s “Concept of Operations for Managing Resource Requests,” we selected a random, statistical⁶ sample of 60 COVID-19 resource requests from the total population of 6,460 COVID-19 resource requests completed during the audit period, using a 95% confidence level,⁷ a 0% expected error rate,⁸ and a 5% tolerable error rate.⁹

For our audit purposes, we defined accurately and timely as detailed below, based upon MEMA’s Concept of Operations for Managing Resource Requests procedures. This document outlined MEMA’s processes on how it fulfilled resource requests. We confirmed these definitions with MEMA management.

“Accurately” was defined as whether the amount and type of the resource(s) delivered equaled the amount and type of resource(s) requested. If the resource(s) requested were unavailable at the time of the request, then substitutions were allowed if agreed upon between MEMA and the requester.

MEMA’s definition of “timely” comprised the following three criteria:

1. Appropriateness of the request was based on whether MEMA fulfilled the request according to the COVID-19 Resource Command Center’s (CCC’s) Resource Flow document. For example, MEMA was responsible for fulfilling requests from local emergency management directors, local fire departments, local county governments, state agencies, and similar organizations. DPH was responsible for fulfilling requests from hospitals, nursing homes, and other similar healthcare organizations. Appropriateness was also based on whether the resource request was related to COVID-19, (e.g., personal protective equipment (PPE) or cleaning supplies).
2. Urgency of the request was based on priority level (Urgent, High, Medium, or Low), which was modified during the COVID-19 pandemic from the Concept of Operations for Managing Resource Requests procedures to the following:
   • Urgent: 1–3 days
   • High: 3–7 days
   • Medium: 7–10 days
   • Low: 10+ days
3. Availability of the resource was defined as the amount of the resource requested in MEMA’s inventory balanced against other incoming requests and MEMA’s ability to provide the resource to the requester, either by delivering it or by having the requester pick it up from MEMA.

For each of the 60 COVID-19 resource requests in our sample, we performed the following procedures:

• To determine whether MEMA fulfilled COVID-19 resource requests accurately, we compared the total number and type of resources that MEMA delivered to what was requested, allowing for substitutions as agreed upon between MEMA and the requester. If there were substitutions, we confirmed that the reasons for the substitutions were documented.
• To determine whether MEMA fulfilled COVID-19 resource requests according to the criteria we used to define timely for this testing (appropriateness, urgency, and availability), we took the following actions:
  • For appropriateness, we inspected the requester’s information on MEMA’s resource request detail form to confirm that MEMA was responsible for fulfilling the resource request and that the resources requested were related to COVID-19.  
  • For urgency, we confirmed that the assigned priority aligned with the requester’s due date and, based on the assigned priority, we confirmed that the fulfillment of the request met the required timing.
  • For availability, we identified whether the resource requests were fully or partially completed. For the requests that were partially completed, we confirmed that the requester agreed to either a substitute resource or fulfillment of the partial resource request.

Additionally, from a population of 719 proactive COVID-19 resource requests¹⁰ that were made when MEMA had an excess of PPE during the audit period, we selected a random, nonstatistical¹¹ sample of 50 proactive COVID-19 resource requests. For each of the 50 proactive COVID-19 resource requests in our sample, we performed the actions described below.

To determine whether MEMA fulfilled proactive COVID-19 resource requests accurately, we compared the total number and type of resources MEMA delivered to what was requested in the Smartsheet.¹²

To determine whether MEMA fulfilled proactive COVID-19 resource requests in a timely manner, we verified that MEMA fulfilled proactive COVID-19 resource requests appropriately (as defined above) by comparing the requester information in the Smartsheet to the Resource Flow document. We only tested for appropriateness because there was no question of urgency or availability, as these were proactive resource requests. Resources were readily available, and we did not measure urgency.

We noted no significant exceptions in our testing. Therefore, we concluded that MEMA fulfilled COVID-19 resource requests accurately and in a timely manner during the audit period. 

MEMA’s Assistance with the FEMA Public Assistance Grant Program

We determined whether MEMA assisted applicants with FEMA Public Assistance Grant Program applications related to COVID-19 in accordance with Section IV (A) numbers 1, 8, 12, 13, 14, 17, and 21 of MEMA’s “State Administration Plan for FEMA Public Assistance Grant Program” for calendar year 2023,¹³ which states in part the following:

[MEMA] shall provide guidance and assistance, including, but not limited to:

1.   Submitting the Application for Federal Assistance (SF-424). . . .

8.   Conducting Applicant Briefings and attending Recovery Scoping Meetings. . . .

12.  Assisting FEMA in developing project worksheets, determining work and applicant eligibility, and identifying special considerations.

13.  Approving project worksheets prior to final obligation. . . .

14.  Notifying subrecipient or applicants of decisions regarding time extensions . . .

17.  Reviewing and certifying project completion information. . . .

21.  Closing out project worksheets (PWs), applicants, and programs.

To do this, we selected a random, statistical sample of 60 COVID-19-related FEMA Public Assistance Grant Program projects from a population of 3,315 COVID-19-related FEMA Public Assistance Grant Program projects from the audit period, using a 95% confidence level, a 0% expected error rate, and a 5% tolerable error rate. For each of the COVID-19-related Public Assistance Grant Program projects selected, we inspected supporting documentation, including requests for public assistance applications, the notes from Recovery Scoping Meetings, MEMA checklists for pre- and post-reviews of FEMA’s project worksheet, project worksheets, payments to the applicants and time extensions (FEMA issued a blanket time extension to applicants for all projects).

We noted no exceptions in our testing. Therefore, we concluded that, during the audit period, MEMA assisted applicants with COVID-19-related FEMA Public Assistance Grant Program applications in accordance with Section IV(A) numbers 1, 8, 12, 13, 14, 17, and 21 of MEMA’s “State Administration Plan for the FEMA Public Assistance Grant Program [Calendar Year] 2023.”

MEMA Collaboration with DPH and OSD

To determine to what extent MEMA collaborated with OSD and DPH in coordinating COVID-19 resource management and delivery, we performed the procedures described below.

CCC Membership

We judgmentally selected 68 CCC situation reports, called Sitreps, out of a total population of 140 Sitreps that were issued during the audit period. We targeted all 45 Sitreps that were issued daily during the first two months of the COVID-19 pandemic, which covered the period March 17, 2020 through April 30, 2020. CCC issued the remaining reports, which covered the 14-month period from May 1, 2020 through June 17, 2021, on a more sporadic basis. From these remaining reports, we selected a judgmental, nonstatistical sample of 23 Sitreps—at least one Sitrep from each month. Our sample included multiple reports issued during the months of September 2020, October 2020, November 2020, and December 2020. We inspected the 68 Sitreps for evidence of collaboration between MEMA and DPH and/or OSD (e.g., mentions of MEMA, DPH, and/or OSD working on the same taskforce; distribution of PPE by MEMA or DPH; and the establishment of the interagency working group).

Interagency Meetings

• We inspected all 74 (100%) PowerPoint presentations documenting virtual meetings between MEMA, DPH, and OSD on COVID-19 inventory status and resource delivery. The presentations covered the period May 22, 2020 through March 1, 2022.
• We inspected all 117 emails provided to us by MEMA that were exchanged between MEMA, OSD, the Executive Office of Health and Human Services, DPH, vendors, consultants, and other state agencies regarding resource management, inventory movement, and demand forecasting. The emails covered the period March 23, 2020 through May 8, 2023.
• We examined Microsoft Teams screenshots showing a meeting that took place on July 22, 2020 between MEMA and OSD regarding personal protective equipment (PPE) and five weekly meetings between MEMA and DPH regarding PPE covering the period March 29, 2022 through April 26, 2022.

Intergovernmental Service Agreements

We examined the interdepartmental service agreement (ISA) between DPH and MEMA, effective December 7, 2020 through June 30, 2021, for providing COVID-19-related clinical supplies and PPE to local health departments, emergency departments, and community health centers.

Collaborative Initiatives

We examined MEMA Situation Report #4, dated March 16, 2020, which established the Interagency Working Group. We examined all 28 Interagency Working Group Sub-Committee Reports that showed evidence of collaboration between MEMA and DPH between March 13, 2020 and April 30, 2020.

We examined the Massachusetts COVID-19 Vaccination Plan (Interim Draft) from DPH, dated October 16, 2020, with MEMA participating as a member of the Leadership and Communications Group.

We examined the press release, dated August 7, 2020, regarding MEMA’s role in the COVID-19 Enforcement and Intervention Team, and we examined an email from MEMA, dated August 28, 2024, which demonstrated evidence of its collaboration with this team.

Social Media

We examined MEMA’s Facebook posts that showed its collaboration with DPH (e.g., mentions of how MEMA and DPH collaboratively worked to procure emergency materials and equipment in response to the COVID-19 pandemic) during the audit period.

We noted no exceptions in our testing. Therefore, we concluded that, during the audit period, MEMA collaborated with OSD and DPH in coordinating COVID-19 resource management and delivery.

Receipt, Expense, and Reporting of COVID-19 Funds

To determine whether MEMA had controls in place over the receipt, expense, and reporting of COVID-19 funds in accordance with MEMA’s reimbursement and payment practices, we performed the following procedures.

Receipt

We selected a random, statistical sample of 60 COVID-19 revenue transactions from a population of 3,002 COVID-19 revenue transactions in the Massachusetts Management Accounting and Reporting System (MMARS) from the audit period, using a 95% confidence level, a 0% expected error rate, and a 5% tolerable error rate. We tested the COVID-19 revenue transactions to ensure that a supervisor approved the revenue that MEMA received.

For each of the samples selected, we inspected the following documents for supervisor approvals:

• MMARS Cash Deposit Forms;
• MMARS Cash Receipts showing transfers to the Commonwealth’s General Fund, where applicable; and
• Cash Management Improvement Act of 1990 Forms.

Expense

We selected a random, statistical sample of 108 expenditures from a population of 19,909 COVID-19 expenditures in MMARS from the audit period, using a 95% confidence level, a 1% expected error rate, and a 5% tolerable error rate. We tested the transactions for supervisor approvals over both the expenditure request and the payment.

For each of the samples selected, we inspected the following documents for supervisor approvals:

• expenditure request—MMARS Payment Request document, MMARS cost allocation report (for payroll-related expenditures), and related invoices and emails approving the expenditure request and
• payment—MMARS electronic funds transfer document and approved timesheets.

Reporting

Based on the terms of the four ISAs with the Executive Office for Administration and Finance (ANF), effective between March 30, 2020 and June 30, 2022, MEMA was required to submit biweekly reports of ISA-related expenditures to ANF. We calculated a total of 55 biweekly reports that MEMA should have submitted to ANF during the audit period. When we requested the 55 biweekly reports, MEMA stated that it did not send biweekly reports to ANF during the audit period. (See Finding 1 for more information.) However, MEMA provided 27 cost reports that it periodically submitted to ANF during the audit period, according to ANF’s request.

Additionally, based on the terms of the ISA with the Department of Public Health (DPH), effective December 7, 2020 through June 30, 2021, MEMA was required to submit a report of expenditures upon termination of the ISA. We inquired whether MEMA submitted the report upon termination of the ISA with DPH and noted that MEMA did not submit the required report of expenditures to DPH. (See Finding 1 for more information.)

See Finding 1 for more information regarding the results of our testing of MEMA’s reporting of COVID-19 funds.

We used both statistical and nonstatistical sampling methods; however, we did not project the results of our testing to any of the corresponding populations.

Data Reliability Assessment

MEMA’s Resource Request System

To determine the reliability of MEMA’s Resource Request System data, we interviewed MEMA’s assistant director of operations and other MEMA employees who were knowledgeable about the system. We tested certain general information system controls, including security management, access controls, configuration management, and contingency planning for MEMA’s Resource Request System to determine the reliability of the data therein.

To determine the completeness and accuracy of the resource request data from the Resource Request System (i.e., the resource requests), we observed the MEMA assistant director of operations query the Resource Request System and extract 6,717 resource requests that were made during the audit period. The assistant director of operations then provided the list of 6,717 resource requests to us in an Adobe PDF, which we converted to a Microsoft Excel spreadsheet. We ensured that the total number of requests we observed within the Resource Request System matched the total number of requests in both the Adobe PDF and the Excel spreadsheet. We identified a total of 234 canceled requests that were made during the audit period and selected a systematic sample¹⁴ of 10 canceled requests to determine whether the canceled requests needed to be included in the final test population. We obtained copies of the Resource Request System’s detailed request forms and reviewed them to understand the reasons for cancelation. Based on the review, we determined that we could exclude canceled requests from the population, thus reducing the population to 6,483 resource requests. We inspected the data for hidden rows and columns, blank fields, and request dates outside the audit period. We also tested the data for duplicate requests (e.g., requests with the same tracking number that were the result of the same request being in process in the first COVID-19 incident and completed in the second COVID-19 incident within MEMA’s Resource Request System) and eliminated 23 duplicates, resulting in a final total population of 6,460 resource requests.

From the population of 6,460 resource requests that occurred during the audit period, we judgmentally selected a sample of 20 resource requests from the Excel spreadsheet and traced the resource request details (e.g., submitted date, tracking number, requester, assistance requested, date needed, requester priority, date fulfilled, and resources deployed) to the resource request detail forms. We followed up with MEMA officials to review required documentation to clarify any anomalies in the data, including, but not limited to, duplicates and blank fields.

Smartsheet Proactive Resource Requests

To determine the reliability of the Smartsheet resource requests, we interviewed MEMA’s assistant director of operations and other MEMA employees who were knowledgeable about the Smartsheet that MEMA used for proactive resource management. MEMA provided the requests in an Excel format for the audit period, which consisted of 719 requests. We tested user access controls. We also verified the location of the requester to determine whether the requester was from a Massachusetts city or town and not from out of state. Additionally, we followed up with MEMA officials to review required documentation to clarify any anomalies in the data, such as hidden rows and columns, blank fields, duplicate requests, and request dates outside the audit period.

FEMA Grants Management System

To determine the reliability of data from FEMA’s Grants Management System, which is used by MEMA to administer FEMA’s Public Assistance Grant Program, we interviewed MEMA employees who were knowledgeable about the system and tested user access controls.

Further, we observed MEMA’s recovery unit supervisor query the system and extract 3,315 COVID-19-related FEMA Public Assistance Grant Program projects that were submitted during the audit period. The recovery unit supervisor then provided the 3,315 COVID-19-related Public Assistance Grant Program projects to us in an Adobe PDF, which we converted to an Excel spreadsheet. We ensured that the total number of COVID-19-related Public Assistance Grant Program projects we observed within the system matched the total number of COVID-19-related Public Assistance Grant Program projects in both the Adobe PDF and the Excel spreadsheet. We followed up with MEMA officials to review required documentation to clarify any anomalies in the data, including, but not limited to, blank fields, duplicate projects, and application dates outside the audit period.

We also compared the data from FEMA’s system of record to the population of 3,315 COVID-19-related Public Assistance Grant Program projects and noted anomalies in the data. We followed up with MEMA officials to review required documentation to clarify any anomalies in the data, including, but not limited to, incomplete data (e.g., recipients without a project number).

Additionally, from the population of 3,315 COVID-19-related Public Assistance Grant Program projects, we judgmentally selected a sample of 20 COVID-19-related Public Assistance Grant Program projects and traced the COVID-19-related Public Assistance Grant Program project details (e.g., project number, project worksheet number, applicant, project type, project title, process step, project size, declaration date, subrecipient type, activity completion deadline, number of damages, best available cost and best available federal share cost) to either the final FEMA project report or screenshots from FEMA’s Grants Management System (for projects either still pending or withdrawn).

MMARS

In 2018 and 2022, the Office of the State Auditor performed a data reliability assessment of MMARS that focused on testing selected system controls (access, security awareness, audit and accountability, configuration management, identification and authentication, and personnel security). We interviewed MEMA employees who were knowledgeable about MMARS, which MEMA used for finance management. We tested certain general information system controls, including security management (i.e., obtained and reviewed security awareness and training policies and procedures, personnel screenings / Criminal Offender Record Information [CORI] background checks, and security training certificates of completion) and access controls (i.e., supervision and review of user access to MMARS), to determine the reliability of the data therein. (See Other Matters for more information regarding the results of our testing of CORI background checks.)

To determine the completeness and accuracy of MEMA’s MMARS data, which detailed its COVID-19 expenditure and revenue transactions, we queried the Commonwealth’s Information Warehouse¹⁵ to obtain a total population of MEMA’s 3,002 COVID-19 revenue transactions and 19,909 COVID-19 expenditures for the audit period. We followed up with MEMA officials to review required documentation to clarify any anomalies in the data, including, but not limited to, blank fields, duplicate records, and data entry errors (i.e., incorrect reporting codes for expenditures).

We judgmentally selected a sample of 20 expenditures from the 19,909 COVID-19 expenditures. For each of the samples selected, we traced the expenditures to at least one of the following documents: (1) invoices, (2) MMARS cost allocation reports, or (3) MMARS payment or request documents to either (a) MMARS electronic funds transfer documents or (b) automated distribution documents. We did this to verify the posting amount, vendor customer number, and vendor payment.

We judgmentally selected a sample of 20 revenue transactions from the 3,002 revenue transactions. For each of the transactions in our sample, we traced the revenue transactions to Cash Management Improvement Act of 1990 draw reports, State Public Assistance Grant Summary, and MEMA-provided central expense details to verify the amount of revenue received.

Based on the results of the data reliability assessment procedures described above, we determined that the information we obtained was sufficiently reliable for the purposes of our audit.

6.    Auditors use statistical sampling to select items for audit testing when a population is large (usually over 1,000) and contains similar items. Auditors generally use a statistical software program to choose a random sample when sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

7.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

8.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

9.    Tolerable error rate is the maximum error in the population that auditors would be willing to accept and still conclude that the result from the sample has achieved the audit objective.

10.    See the “Proactive Resource Request Management” section for an explanation of proactive COVID-19 resource requests.

11.    Auditors use nonstatistical sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review.

12.    Local emergency management directors, local fire departments, county governments, state agencies, and similar organizations could request excess PPE by completing and submitting a data entry form in Smartsheet to MEMA and DPH.

13.    MEMA’s “State Administration Plan for FEMA Public Assistance Grant Program” documents for calendar years 2021 and 2022 include the same requirements as calendar year 2023.

14.    A systematic sample is a sample selected from a population using a fixed interval (e.g., every Nth number).

15.    The Commonwealth Information Warehouse contains historical budget, human resources, and payroll information as well as financial transaction data from the Massachusetts Management Accounting and Reporting System.

Date published:	June 11, 2025