Audit of the Veterans Home at Chelsea Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Veterans Home at Chelsea.

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Veterans Home at Chelsea (VHC) for the period July 1, 2021 through June 30, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings. 

Objective	Conclusion
1. Did VHC have policies and procedures in place to ensure that it had sufficient staffing to meet its veterans’ needs in accordance with Sections 51.110(b) and 51.130 of Title 38 of the Code of Federal Regulations (CFR)?	No; see Findings 1, 2, and 3
2. Did VHC ensure that the need for nursing overtime was documented and approved in accordance with VHC’s Overtime Policy 180-11-2022-A?	No; see Finding 4
3. Did VHC have an emergency operations plan (EOP) in place, and was it updated to address the risk involved in providing adequate patient care during an emergency in accordance with Section 150.015(E) of Title 105 of the Code of Massachusetts Regulations (CMR)?	No; see Findings 5a, 5b, 5c, and 5d

To accomplish our audit objectives, we gained an understanding of the VHC internal control environment relevant to our objectives by reviewing VHC’s policies and procedures, as well as by conducting inquiries with its staff members and management. We evaluated the design of controls over VHC’s management review and approval of overtime and the Overtime Committee and retention of the overtime request forms as part of our substantivate testing. In addition, to obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Review of Staffing

To determine whether VHC had policies and procedures in place to ensure that it had sufficient staffing in accordance with 38 CFR 51.110(b) and 51.130, we met with VHC’s scheduler and director of nursing to gain an understanding of the process. Next, we obtained a list of veterans who were at VHC during the audit period. We selected a nonstatistical,⁶ judgmental sample of 35 veterans from a population of 153. To complete our testing, we reviewed the hardcopy medical records for each veteran in our sample and determined whether each veteran had an individualized comprehensive plan of care⁷ and had a completed assessment on file. Next, we reviewed the plans of care and assessments to determine whether, during the audit period, any of the veterans in our sample had a prescription for 1:1 care or whether the assessment indicated that the veteran was a fall risk. We then judgmentally selected a date for each veteran in our sample for when the assessment indicated they were prescribed 1:1 care or deemed a fall risk and requested the corresponding assignment sheets to determine whether the assessment of each veteran in our sample aligned with the duties assigned to nursing staff members. For example, any veteran who had a 1:1 care prescription in their medical file, we determined whether an extra staff member was scheduled for that veteran on the assignment sheet. If the veteran was deemed a fall risk, we sought to determine whether the assignment sheet reflected this status, so that staff members were made aware of the need to manage this risk.

Based on the results of our testing, VHC was missing the following requested documentation: assignment sheets, veteran assessments, daily staffing schedules, and incident logs. See Findings 1, 2, and 3 for more information.

Review of Overtime

To determine whether VHC documented the need and approval of overtime for nursing staff members in accordance with VHC’s Overtime Policy 180-11-2022-A, we obtained labor history data from the Massachusetts Management Accounting and Reporting System (MMARS). We filtered this data to only include nursing staff members who were paid overtime during the audit period. Nursing staff members included licensed practical nurses, nurse practitioners, certified nursing assistants, and registered nurses. We also filtered the population to only include overtime premium, overtime straight, callback premium, and callback straight pay. This resulted in a population of 5,669 premium and straight overtime and callback occurrences⁸ from which we selected our sample for testing.

We selected a statistical,⁹ random sample of 128 overtime transactions out of the population of 5,669, using a confidence level of 90%,¹⁰ a 15% desired precision range,¹¹ and 50% expected error rate.¹² We reviewed the hardcopy overtime request forms to determine whether the form was submitted for approval, the time requested matched the time paid, and the overtime was approved by a supervisor with their signature, as well as the reason for the overtime.

In addition, we requested evidence of the biweekly Overtime Committee meetings, overtime reports run during the audit period, and any evidence of department heads or supervisors addressing overtime that was not approved. Furthermore, we interviewed staff members to ensure that Overtime Request Forms were properly maintained in accordance with the Massachusetts Statewide Records Retention Schedule.

Based on the results of our testing, VHC did not follow its Overtime Policy 180-11-2022-A in documenting the need for and approval of overtime and conducting biweekly Overtime Committee meetings, evidencing the implementation of monitoring controls. See Finding 4 for more information.

Review of EOP

In order to determine whether VHC had an EOP in place and whether it was updated to address the risk involved in providing adequate patient care during an emergency in accordance with 105 CMR 150.015(E), we obtained and reviewed a copy of VHC’s February 2020 internal control plan, which includes the EOP.

We then performed the following procedures:

• We conducted a tour of VHC to determine whether there was a written plan in place and whether there were copies of the plan posted throughout the facility.
• We reviewed the hardcopy EOP contained in the incident command center to determine whether the plan detailed which people should be notified and when; the location of alarm signals, fire extinguishers, and evacuation routes; procedures for evacuation of veterans; and assignment of responsibilities to the personnel of each shift.
• We reviewed After Action Reports to determine whether staff members conducted a minimum of two drills per year per shift.
• We reviewed After Action Reports to determine whether a reliable means of communication was always available for sending and receiving information from the Department of Public Health.

Based on the results of our testing, VHC’s EOP did not meet the requirements of 105 CMR 150.015(E). See Findings 5a, 5b, 5c, and 5d for more information.

We used a combination of statistical and nonstatistical sampling methods for testing. Where we used nonstatistical sampling methods, we did not project the results of our testing to the corresponding population.

Data Reliability Assessment

In 2022, the Office of the State Auditor performed data reliability assessments of the Massachusetts Management Accounting and Reporting System (MMARS) focused on testing selected system controls (access, security awareness, audit and accountability, configuration management, identification and authentication as well as personnel security). In addition, as part of our current audit, we tested certain general information system controls, including security management (i.e., obtained and reviewed security awareness and training policies and procedures, personnel screenings/Criminal Offender Record Information background checks, and security training certificates of completion) and access controls (i.e., supervision and review of user access to MMARS) to determine the reliability of the data therein. We also conducted tests to identify any duplicates, a criteria check to ensure that the data was within the audit period, and a criteria check for any blanks in pertinent information. Further, we selected a random sample of 20 overtime occurrences from the MMARS Labor History and vouched¹³ these occurrences to physical documentation (overtime request forms). We also selected a sample of 20 overtime request forms from VHC’s files and traced these files to the MMARS Labor History. See Finding 4 for more information regarding the results of our overtime request test.

To determine the reliability of the data from the list of all residents at VHC during the audit period, we vouched a sample of 20 residents from the list to their resident files and selected 20 resident files that we traced back to the list. In addition, we conducted tests to identify any duplicates to determine the integrity of the information on the list.

Based on the results of the data reliability assessment procedures described above, we determined that all data we obtained during the course of our audit was sufficiently reliable for the purposes of our audit.

6. Auditors use nonstatistical sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review.
7. An individualized plan of care is a document that outlines an individual’s specific health conditions, goals, necessary services, and the treatments needed to meet their goals.
8. We included incremental overtime within our population of premium, straight, and callback overtime.
9. Auditors use statistical sampling to select items for audit testing when a population is large and contains similar items. Auditors generally use a statistical software program to choose a random sample when sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.
10. Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.
11. Desired precision range is the range of likely values within which the true population value should lie; also called confidence interval. For example, if the interval is 90%, the auditor will set an upper confidence limit and a lower confidence where 90% of transactions fall within those limits.
12. Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.
13. Vouching is the inspection of supporting documentation to corroborate data.

Date published:	January 14, 2026