Audit of Cybersecurity Awareness Training Compliance Across Multiple State Agencies Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Audit of Cybersecurity Awareness Training Compliance Across Multiple State Agencies.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of cybersecurity awareness training at the Executive Office of Technology Services and Security (EOTSS). Pursuant to our governing statute, Section 12 of Chapter 11 of the General Laws, our audit covers multiple entities’ compliance with EOTSS’s cybersecurity training standards. Specifically, Section 12 of Chapter 11 states, “Each entity may be audited separately as a part of a larger organizational entity or as a part of an audit covering multiple entities.” As such, cybersecurity awareness training testing was completed at 22 other executive branch agencies, state colleges and universities, and regional transit authorities, for the period July 1, 2021 through April 30, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Below is our audit objective, indicating the question we intended our audit to answer; the conclusion we reached regarding our objective; and, if applicable, where our objective is discussed in the audit findings.

Objective	Conclusion
Did EOTSS and other executive branch agencies, state colleges and universities, and regional transit authorities ensure that their employees completed cybersecurity awareness training in accordance with Sections 6.2.3 and 6.2.4 of EOTSS’s Information Security Risk Management Standard IS.010?	No; see Findings 1, 2, 3, and 4

To accomplish our audit objective, we gained an understanding of the aspects of EOTSS’s internal control environment relevant to our objective by interviewing EOTSS staff members and management and by reviewing EOTSS’s Information Security Risk Management Standard IS.010.

To obtain sufficient, appropriate evidence to address our audit objective, we performed the procedures described below.

Cybersecurity Awareness Training

We separated the 23 agencies we reviewed as part of this audit into three categories based on agency type: EOTSS and other executive branch agencies, state colleges or universities, and regional transit authorities.

• The first category comprises EOTSS and 11 other executive branch agencies: the Bureau of the State House (BSH), the Civil Service Commission (CSC), the Department of Labor Standards (DLS), the Department of Mental Health (DMH), the Department of Public Health (DPH), the Department of Revenue (DOR), the Massachusetts Department of Transportation (MassDOT), the Group Insurance Commission (GIC), the Massachusetts Parole Board (MPB), the Registry of Motor Vehicles (RMV), and the State 911 Department (911).
• The second category comprises seven state colleges and universities: Framingham State University (FSU), Holyoke Community College (HCC), Massachusetts Bay Community College (MBCC), Massasoit Community College (MCC), North Shore Community College (NSCC), Northern Essex Community College (NECC), and Westfield State University (WSU).
• The third category comprises four regional transit authorities: the Cape Ann Transportation Authority (CATA), the Cape Cod Regional Transit Authority (CCRTA), the Martha’s Vineyard Regional Transit Authority (VTA), and the Nantucket Regional Transit Authority (NRTA).

To determine whether EOTSS and these other executive branch agencies, state colleges and universities, and regional transit authorities ensured that their employees completed cybersecurity awareness training in accordance with Sections 6.2.3 and 6.2.4 of EOTSS’s Information Security Risk Management Standard IS.010, we took the actions described below.

EOTSS and Other Executive Branch Agencies

To determine whether EOTSS and the 11 other executive branch agencies included in this audit ensured that their newly hired employees completed initial cybersecurity awareness training within 30 days of orientation, we analyzed the evidence for cybersecurity awareness training completion (i.e., transcript reports⁵) by comparing each employee’s start date and training completion date for all 2,662 newly hired employees across these executive branch agencies.

To determine whether these executive branch agencies ensured that their existing employees completed annual refresher cybersecurity awareness training, we analyzed the evidence for cybersecurity awareness training completion (i.e., transcript reports) by comparing each employee’s training completion date and training due date for all 12,236 existing employees across these executive branch agencies.

To further substantiate the results of the above procedures, we also selected a random, statistical sample⁶ of 24 employee training certificates of completion out of the population of 14,898 newly hired and existing employees, using a 90% confidence level,⁷ a 0% expected error rate,⁸ and a 10% tolerable error rate.⁹ Our sample comprised the following:

• from EOTSS, BSH, CSC, DLS, GIC, MPB, RMV, and 911: 1 employee training certificate of completion from each agency;
• from DOR: 2 employee training certificates of completion;
• from DPH and MassDOT: 4 employee training certificates of completion from each agency; and
• from DMH: 6 employee training certificates of completion.

We selected these sample numbers based on the number of active employees each agency had during the audit period.

We did not note any exceptions in our testing corresponding to BSH and 911. Therefore, we concluded that, during the audit period, BSH and 911 met the relevant criteria regarding this matter.

For the other executive branch agencies included in this audit, we did note exceptions during our testing. See Findings 1 and 2 for issues we identified with the cybersecurity awareness training provided by EOTSS and the other executive branch agencies included in this audit.

State Colleges and Universities

To determine whether the state colleges and universities included in this audit ensured that their employees completed cybersecurity awareness training, we took the actions described below.

We inspected the cybersecurity awareness training certificates of completion using a judgmental,¹⁰ nonstatistical sample of 70 employee training certificates of completion out of the population of 10,094. Our sample comprised 10 employee training certificates of completion from each of the seven state colleges and universities included in this audit. Of the 10 employee training certificates of completion from each state college or university, we judgmentally selected 3 existing non-student employees, 4 newly hired non-student employees, and 3 existing student employees.

Also, we determined whether the state colleges and universities included in this audit ensured that the newly hired employees from our sample completed initial training within 30 days of orientation by comparing the dates of their orientations to the dates of their certificates of completion.

See Finding 3 for issues we identified with the cybersecurity awareness training provided by the state colleges and universities included in this audit.

Regional Transit Authorities

To determine whether the regional transit authorities included in this audit ensured that their employees completed cybersecurity awareness training, we took the actions described below.

We inspected the cybersecurity awareness training certificates of completion using a judgmental, nonstatistical sample of 23 employee training certificates of completion out of the population of 55. Our sample comprised the following:

• from CATA: 3 employee training certificates of completion (which represents its full population of employees);
• from NRTA: 4 employee training certificates of completion (which represents its full population of employees); and
• from CCRTA and VTA: 8 employee training certificates of completion from each agency.

Of the 8 employee training certificates of completion from CCRTA and VTA, we judgmentally selected 2 newly hired employees and 6 existing employees. Additionally, we determined whether these regional transit authorities ensured that the newly hired employees from our sample completed initial training within 30 days of orientation by comparing the dates of their orientations to the dates of their certificates of completion.

We did not note any exceptions in our testing corresponding to NRTA. Therefore, we concluded that, during the audit period, NRTA met the relevant criteria regarding this matter.

For the other regional transit authorities included in this audit, we noted exceptions during our testing. See Finding 4 for issues we identified with the cybersecurity awareness training provided by the regional transit authorities included in this audit.

We used a combination of statistical and nonstatistical sampling methods for testing, and we did not project the results of our testing to any corresponding populations.

Data Reliability Assessment

To determine the reliability of the employee lists from EOTSS and each of the 22 other executive branch agencies, state colleges and universities, and regional transit authorities included in this audit (see the list of auditees included in this report, by category), we took the actions described below.

We interviewed EOTSS management who were knowledgeable about these lists. We reviewed MassAchieve¹¹ system controls for access control, configuration management, contingency planning, segregation of duties, and security management. We checked that the variable formats of each agency’s employee list (e.g., dates, unique identifiers, or abbreviations) were accurate. For each agency’s employee list, we ensured that there was no abbreviation of data fields, no missing data (e.g., hidden rows or columns, blank cells, or incomplete records), and no duplicate records and that all values corresponded with expected values.

To determine the completeness and accuracy of each agency’s employee list, we took the actions described below.

EOTSS and Other Executive Branch Agencies

• EOTSS: We selected random samples of 20 employees from EOTSS’s employee list and traced their names to CTHRU, the Commonwealth’s statewide payroll open records system. We also selected random samples of 20 employees from CTHRU and traced their names back to EOTSS’s employee list.
• BSH and CSC: We selected random samples of five employees from each executive branch agency’s employee list and traced their names to CTHRU. We also selected random samples of five employees from each agency from CTHRU and traced their names back to each agency’s employee list.
• DLS, GIC, MPB, and 911: We selected random samples of 10 employees from each executive branch agency’s employee list and traced their names to CTHRU. We also selected random samples of 10 employees from each agency from CTHRU and traced their names back to each agency’s employee list.
• DMH, DPH, DOR, MassDOT, and RMV: We selected random samples of 20 employees from each executive branch agency’s employee list and traced their names to CTHRU. We also selected random samples of 20 employees from each agency from CTHRU and traced their names back to each agency’s employee list.

State Colleges and Universities

• FSU, HCC, MBCC, MCC, NSCC, NECC, and WSU: We selected random samples of 20 employees from each state college’s/university’s employee list and traced their names to CTHRU. We also selected random samples of 20 employees from each state college/university from CTHRU and traced their names back to each state college/university’s employee list.

Regional Transit Authorities

• CATA: We selected the total population of three employees and traced their names to CATA’s open payroll webpage. We also selected the total population of three employees from CATA’s open payroll webpage and traced their names back to CATA’s employee list.
• CCRTA and VTA: We selected random samples of five employees from each regional transit authority’s employee list and traced their names to each agency’s open payroll webpage. We also selected random samples of five employees from each regional transit authority’s open payroll webpage and traced their names back to each agency’s employee list.
• NRTA: We selected the total population of four employees and traced their names to NRTA’s open payroll webpage. We also selected the total population of four employees from NRTA’s open payroll webpage and traced their names back to NRTA’s employee list.

Based on the results of the data reliability assessment procedures described above, we determined that the information we obtained for the audit period was sufficiently reliable for the purposes of our audit.

5.    We analyzed the cybersecurity awareness training transcript reports from EOTSS and the other executive branch agencies. These reports included fields such as the training due date and the training completion date.

6.    Auditors use statistical sampling to select items for audit testing when a population is large and contains similar items. Auditors generally use a statistical software program to choose a random sample when sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

7.   Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

8.   Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

9.   The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

10.    Auditors use judgmental sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review. Auditors use their knowledge and judgment to select the most appropriate sample. For example, an auditor might select items from areas of high risk. The results of testing using judgmental sampling cannot be used to make conclusions or projections about entire populations; however, they can be used to identify specific issues, risks, or weaknesses.

11.   MassAchieve is a training platform used by executive branch agencies to administer cybersecurity awareness training.

Date published:	November 8, 2024