Audit of the MassHire Department of Career Services Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the MassHire Department of Career Services.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the MassHire Department of Career Services (MDCS) for the period July 1, 2022 through June 30, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Was MDCS’s website in compliance with the Executive Office of Technology Services and Security’s (EOTSS’s) Enterprise Information Technology Accessibility Policy and the World Wide Web Consortium’s (W3C’s) Web Content Accessibility Guidelines (WCAG) 2.1 for user accessibility, keyboard accessibility, navigation accessibility, language, error identification, and color accessibility?	No; see Findings 1 and 2
Was the JobQuest website in compliance with EOTSS’s Enterprise Information Technology Accessibility Policy and W3C’s WCAG 2.1 for user accessibility, keyboard accessibility, navigation accessibility, language, error identification, and color accessibility?	No; see Finding 3
Did MDCS do the following to implement certain information technology (IT) governance policies: establish classification or sensitivity levels of all information of which it had custody in accordance with Section 6.2. of EOTSS’s Asset Management Standard IS.004; identify and securely delete stored information that exceeded defined retention periods on a quarterly basis in accordance with Section 6.4.2.4. of EOTSS’s Asset Management Standard IS.004; and conduct a business impact analysis or risk assessment to determine the classification level of information systems in accordance with Section 6.6.2. of EOTSS’s Asset Management Standard IS.004?	No; see Findings 4, and 5
Did MDCS restrict access to personally identifiable information (PII) to a narrow subset of personnel members who had a business need to access the information in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004?	No; see Findings 6 and 7

To accomplish our audit objectives, we gained an understanding of MDCS’s internal control environment relevant to our objectives by reviewing applicable policies and procedures and by interviewing MDCS staff members and management. In addition, to obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Accessibility Testing on MDCS’s and Career Centers’ Websites

To determine whether MDCS’s website and the websites of the career centers were in compliance with EOTSS’s Enterprise Information Technology Accessibility Policy and W3C’s WCAG 2.1 for user accessibility, keyboard accessibility, navigation accessibility, language, error identification, and color accessibility, we took the actions described below.

We reviewed MDCS’s website and 5 career center websites from a total of 18 career center websites. The selected career center websites were those of the following institutions: the Bristol Career Center, the Greater Brockton Career Center, the Greater New Bedford Career Center, the Lowell Career Center, and the North Central Career Center.

To determine whether MDCS’s and the selected career centers’ websites met WCAG 2.1 for user accessibility, we selected random, nonstatistical samples of the following websites:

• MDCS’s website: We selected a sample of 60 webpages out of a total population of 842.
• The Bristol Career Center’s website: We selected a sample of 5 webpages out of a total population of 18.
• The Greater Brockton Career Center’s website: We selected a sample of 10 webpages out of a total population of 28.
• The Greater New Bedford Career Center’s website: We selected a sample of 5 webpages out of a total population of 7.
• The Lowell Career Center’s website: We selected a sample of 20 webpages out of a total population of 59.
• The North Central Career Center’s website: We selected a sample of 10 webpages out of a total population of 39.

We performed the procedures described below on the sampled webpages.

User Accessibility

• We determined whether content on the website was able to be viewed in both portrait and landscape modes.
• We determined whether content on the webpage was undamaged and remained readable when zoomed to 200% and 400%.

Keyboard Accessibility

• We determined whether all elements⁷ of the webpage could be navigated using only a keyboard.
• We determined whether any elements on the webpage prevented a user from moving to a different element when using only a keyboard to navigate the webpage.
• We determined whether the first focusable control⁸ is a hyperlink that redirects to the main content of the website. The first focusable control is known as either a bypass block or a skip link.

Navigation Accessibility

• We determined whether the website contained a title that was relevant to website content.
• We determined whether there was a search function present to help users locate content.
• We determined whether related hyperlinks allowed navigation to the intended webpage.
• We determined whether headings within websites related to the content of the header’s section.

Language

• We determined whether video content found within the website had all important sounds and dialogue captioned.
• We determined whether the language of the webpage was tagged with the correct language attribute.⁹
• We determined whether words that appeared on the webpage matched the language to which the webpage was set.

Error Identification

• We determined whether mandatory form fields alerted users if the field was left blank.
• We determined whether there was a label for elements that required user input.
• We determined whether the label was programmed correctly.
• We determined whether there were examples given to assist the user in correcting mistakes (for example, a warning when entering a letter in a field meant for numbers).

Color Accessibility

• We determined whether there was at least a 3:1 contrast in color and additional visual cues to distinguish hyperlinks, which WCAG recommends for users with colorblindness or other visual impairments.

See Finding 1 for issues we identified with MDCS’s website. See Finding 2 for issues we identified with the career centers’ websites.

JobQuest Website Accessibility Testing

To determine whether the JobQuest website met WCAG 2.1 for user accessibility, we took the actions described below. We inspected the following sample of webpage types from JobQuest’s website:

• JobQuest’s main webpages: We selected a random, nonstatistical sample of 20 webpages out of a total population of 72.
• Job webpages: We selected a random, statistical¹⁰ sample of 60 webpages out of a total population of 2,424 using a 95% confidence level,¹¹ a 0% expected error rate,¹² and a 5% tolerable error rate.¹³
• Training webpages: We selected a random, nonstatistical sample of 60 out of a total population of 840 webpages for trainings that had occurred recently or were upcoming at the time of our testing. Training webpages are removed after the training occurs, so there were no webpages available that were active during the audit period.
• Workshop webpages: We selected a random, statistical sample of 60 out of a total of 2,902 webpages for workshops that had occurred recently or were upcoming at the time of our testing, using a 95% confidence level, a 0% expected error rate, and a 5% tolerable error rate. Workshop webpages are removed after the workshop occurs, so there were no webpages available that were active during the audit period.

On the sampled webpages, we performed the procedures described below.

User Accessibility

• We determined whether content on the website was able to be viewed in both portrait and landscape modes.
• We determined whether content on the webpage was undamaged and remained readable when zoomed to 200% and 400%.

Keyboard Accessibility

• We determined whether all elements of the webpage could be navigated using only a keyboard.
• We determined whether any elements on the webpage prevented a user from moving to a different element when using only a keyboard to navigate the webpage.
• We determined whether the first focusable control is a hyperlink that redirects to the main content of the website. The first focusable control is known as either a bypass block or a skip link.

Navigation Accessibility

• We determined whether the website contained a title that was relevant to website content.
• We determined whether there was a search function present to help users locate content.
• We determined whether related hyperlinks allowed navigation to the intended webpage.
• We determined whether headings within websites related to the content of the header’s section.

Language

• We determined whether video content found within the website had all important sounds and dialogue captioned.
• We determined whether the language of the webpage was tagged with the correct language attribute.
• We determined whether words that appeared on the webpage matched the language to which the webpage was set.

Error Identification

• We determined whether mandatory form fields alerted users if the field was left blank.
• We determined whether there was a label for elements that required user input.
• We determined whether the label was programmed correctly.
• We determined whether there were examples given to assist the user in correcting mistakes (for example, a warning when entering a letter in a field meant for numbers).

Color Accessibility

• We determined whether there was at least a 3:1 contrast in color and additional visual cues to distinguish hyperlinks, which WCAG recommends for users with colorblindness or other visual impairments.

See Finding 3 for issues we identified with the JobQuest website.

IT Governance Testing

We took the following actions to determine whether MDCS established IT governance policies and procedures over the areas listed below.

Information Classification Policy

To determine whether MDCS’s information classification policy met the requirements of Section 6.2 of EOTSS’s Asset Management Standard IS.004, we interviewed knowledgeable MDCS staff members and requested MDCS’s information classification policy. We learned that MDCS did not have an information classification policy in place during the audit period.

See Finding 4 regarding MDCS’s information classification policy.

Information Disposal Plan and Procedures

To determine whether MDCS’s information disposal procedures met the requirements of Section 6.4.2.4. of EOTSS’s Asset Management Standard IS.004, we interviewed knowledgeable MDCS staff members and requested MDCS’s information disposal plan and procedures.

Business Impact Analysis or Risk Assessment to Determine Information System Classification

To determine whether MDCS conducted a business impact analysis or risk assessment in accordance with the requirements of Section 6.6.2. of EOTSS’s Asset Management Standard IS.004, we interviewed knowledgeable MDCS staff members and requested the MDCS business impact analysis or risk assessment used to determine the classification level of MDCS’s information systems. We were informed that MDCS did not conduct a business impact analysis or risk assessment to determine the classification level of its information systems.

See Finding 5 regarding MDCS’s business impact analysis and/or risk assessment.

Restricted Access to PII

To determine whether MDCS and the career centers restricted access to PII to the narrow subset of personnel members who had a business need to access the information in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004, we took the actions described below. We reviewed MDCS and 10 career centers from a total of 28 career centers. The 10 career centers selected were those of the following areas: Berkshire, Greater Brockton, Boston, Cape and Islands, Framingham, Holyoke, Merrimack Valley, Metro North-Woburn, North Central, and North Shore Youth Center.

We requested that knowledgeable MDCS staff members identify personnel members on the MDCS employee list who had access to MDCS’s centralized database. We then requested that knowledgeable staff members at each of the selected career centers identify personnel members on each of their employee lists who had access to MDCS’s centralized database. If a user was listed on both the MDCS employee list and one of the career centers’ employee lists, we included them only in the population for MDCS.

Additionally, for the career centers that stored PII outside of the centralized database, we requested that knowledgeable personnel members at these career centers identify the personnel members on their employee lists who would have had access to it.

We organized our sample consisting of MDCS and the career centers into two subsections. The first subsection contained MDCS and the Greater Brockton Career Center, which only store data in MDCS’s centralized database. For this first subsection, we conducted one test, which involved our inspection of user authorization forms for access to the MDCS centralized database. The second subsection contained the career centers that stored data in both MDCS’s centralized database and their corresponding career center. For the second subsection, we conducted two tests. First, we inspected user authorization forms for access to MDCS’s centralized database, and then we inspected authorization forms for access to PII stored at the corresponding career center. We selected random, nonstatistical samples for the populations below, except for the Boston Career Center, the Metro North–Woburn Career Center, and the North Shore Youth Center. For the Boston Career Center and the Metro North–Woburn Career Center, we selected a random, nonstatistical sample for the first test and tested the whole population for the second test. For the North Shore Youth Center, we tested the whole population for the first test and selected a random, nonstatistical sample for the second test.

First Subsection: MDCS and the Greater Brockton Career Center

• MDCS:

• We inspected the database approval forms for a sample of 35 employees with access to MDCS’s centralized database, from a population of 200 employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.

• Greater Brockton Career Center:

• We inspected the database approval forms for a sample of 10 employees with access to PII in MDCS’s centralized database, from a population of 26 employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.

Second Subsection: Other Career Centers

• Berkshire Career Center:

• We inspected the database approval forms for a sample of 10 employees with access to MDCS’s centralized database, from a population of 26 employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• Berkshire Career Center management informed us that they did not have an authorization process for personnel members to be granted access to PII stored outside of MDCS’s centralized database. Instead, we inspected general confidentiality forms signed at hire for a sample of 5 employees with access to PII stored outside of MDCS’s centralized database, from a population of 12 employees.

• Boston Career Center:

• We inspected the database approval forms for a sample of 10 employees with access to PII in MDCS’s centralized database, from a population of 22 employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• Boston Career Center management informed us that they did not have an authorization process for personnel members to be granted access to PII stored outside of MDCS’s centralized database. Instead, we inspected general confidentiality forms signed at hire for the one employee with access to PII stored outside of MDCS’s centralized database.

• Cape and Islands Career Center:

• We inspected the database approval forms for a sample of 5 employees, from a population of 17 employees with access to PII in MDCS’s centralized database, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• Cape and Islands Career Center management informed us that they did not have an authorization process for personnel members to be granted access to PII stored outside of MDCS’s centralized database. Instead, we inspected general confidentiality forms signed at hire for a sample of 10 employees with access to PII stored outside of MDCS’s centralized database, from a population of 23 employees.

• Framingham Career Center:

• We inspected the database approval forms for a sample of 10 employees with access to PII in MDCS’s centralized database, from a population of 24 employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• Framingham Career Center management informed us that they did not have an authorization process for personnel members to be granted access to PII stored outside of MDCS’s centralized database. Instead, we inspected general confidentiality forms signed at hire for a sample 10 employees with access to PII stored outside of MDCS’s centralized database, from a population of 32 employees.

• Holyoke Career Center:

• We inspected the database approval forms for a sample of 20 employees with access to PII in MDCS’s centralized database, from a population of 52 employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• We inspected the user access forms for a sample 10 employees with access to PII, from a population of 24 employees with access to PII stored outside of MDCS’s centralized database, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.

• Merrimack Valley Career Center:

• We inspected the database approval forms for a sample of 10 employees with access to PII in MDCS’s centralized database, from a population of 38 employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• Merrimack Valley Career Center management informed us that they did not have an authorization process for personnel members to be granted access to PII stored outside of MDCS’s centralized database. Instead, we inspected the general confidentiality forms signed at hire for a sample of 20 employees with access to PII stored outside of MDCS’s centralized database, from a population of 54 employees.

• Metro North-Woburn Career Center:

• We inspected the database approval forms for a sample of 5 employees with access to PII in MDCS’s centralized database, from a population of 18 employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• Metro North-Woburn Career Center management informed us that they did not have an authorization process for personnel members to be granted access to PII stored outside of MDCS’s centralized database. Instead, we inspected general confidentiality forms signed at hire for all three employees with access to PII stored outside of MDCS’s centralized database.

• North Central Career Center:

• We inspected the database approval forms for a sample of five employees with access to PII in MDCS’s centralized database, from a population of eight employees, and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• North Central Career Center management informed us that they did not have an authorization process for personnel members to be granted access to PII stored outside of MDCS’s centralized database. Instead, we inspected the general confidentiality forms signed at hire for a sample of five employees with access to PII stored outside of MDCS’s centralized database, from a population of eight employees.

• North Shore Youth Center:

• We inspected the database approval forms for all four employees with access to PII in MDCS’s centralized database and verified that the employees’ access to PII was approved in accordance with Section 6.2.1. of EOTSS’s Asset Management Standard IS.004.
• North Shore Youth Center management informed us that they did not have an authorization process for personnel members to be granted access to PII stored outside of MDCS’s centralized database. Instead, we inspected the general confidentiality forms signed at hire for a sample of five employees with access to PII stored outside of MDCS’s centralized database, from a population of nine employees.

See Finding 6 regarding MDCS’s authorization process for access to PII. See Finding 7 regarding the career centers’ authorization processes for access to PII.

We used a combination of statistical and nonstatistical sampling methods for testing and did not project the results of our testing to any corresponding populations.

Data Reliability Assessment

Web Accessibility

To determine the reliability of the site map spreadsheets we received from MDCS and each of the five additional career centers, we took the following actions. We interviewed MDCS management, interviewed knowledgeable MDCS staff members, and checked that variable formats (e.g., dates, unique identifiers, or abbreviations) were accurate. Additionally, we ensured that there was no abbreviation of data fields, no missing data (e.g., hidden rows or columns, blank cells, or absent records), no duplicate records, and that all values in the data set corresponded with expected values.

To determine the completeness and accuracy of MDCS’s and each career center’s site maps, we took the following actions:

• MDCS: We selected a random sample of 20 uniform resource locators (URLs)[1] that could be accessed independently from the MDCS site map and traced each to the corresponding webpage, checking that each URL and webpage title matched the information on the MDCS website. We also selected a random sample of 20 URLs from MDCS’s website and traced each URL and webpage title to the site map.
• Greater Lowell: We selected a random sample of 10 URLs that could be accessed independently from the site map and traced them to the corresponding webpage, checking that each URL and webpage title matched the information on the website. We also selected a random sample of 10 URLs from the website and traced each URL and webpage title to the site map.
• Bristol, Greater Brockton, Greater New Bedford, North Central: We selected a random sample of five URLs that could be accessed independently from the site map and traced them to the corresponding webpage, checking that each URL and webpage title matched the information on the website. We also selected a random sample of five URLs from the websites and traced each URL and webpage title to the site map.

JobQuest Web Accessibility

To determine the reliability of the spreadsheets of the main JobQuest webpages, jobs webpages, workshops webpages, and trainings webpages received from MDCS, we took the following actions. We interviewed MDCS management, interviewed knowledgeable MDCS staff members, and checked that variable formats (e.g., dates, unique identifiers, abbreviations) were accurate. Additionally, we ensured that there was no abbreviation of data fields, no missing data (e.g., hidden rows or columns, blank cells, absent records), no duplicate records, and that all values in the data set corresponded with expected values.

In addition, we selected a random sample of 20 URLs that could be accessed independently from each of the spreadsheets and traced them to the corresponding webpage, checking that each URL and webpage title matched the information on the JobQuest website. We also selected a random sample of 20 URLs from JobQuest’s website and traced each URL and webpage title to the site map to ensure that there was a complete and accurate population of URLs on the site map.

IT Governance

To determine the reliability of the employee lists we received from MDCS and each of the 10 career centers, we took the following actions. We interviewed MDCS management and knowledgeable MDCS staff members, and checked that variable formats (e.g., dates, unique identifiers, abbreviations) were accurate. Additionally, we ensured that there was no abbreviation of data fields, no missing data (e.g., hidden rows or columns, blank cells, absent records), no duplicate records, and that all values in the data set corresponded with expected values.

To determine the completeness and accuracy of MDCS’s and each career center’s employee lists, we took the following actions:

• MDCS: We selected random samples of 10 employees from MDCS’s employee list and traced their names to CTHRU, the Commonwealth’s statewide payroll open records system. We also selected a random sample of 10 employees who were listed as MDCS employees on CTHRU and traced their names back to MDCS’s employee list.
• Holyoke and Merrimack Valley: We selected random samples of 10 employees from each employee list and traced their names to pay stubs at the career centers. We also selected random samples of 10 employees from the pay stub records and traced their names back to each employee list.
• Berkshire, Boston, Cape and Islands, Framingham, Greater Brockton, Metro North-Woburn, North Central, and North Shore Youth Center: We selected random samples of five employees from each employee list and traced their names to pay stubs at the career centers. We also selected random samples of 5 employees from the pay stub records and traced their names back to each employee list.

Based on the results of the data reliability assessment procedures described above, we determined that the site maps and employee lists we obtained during the course of our audit were sufficiently reliable for the purposes of our audit.

7.    An element is a part of a webpage that contains data, text, or an image.

8.    The first focusable control is the first element a user will be brought to on a webpage when navigating with a keyboard.

9.    A language tag identifies the native language of the content on the webpage or PDF (e.g., a webpage in English should have an EN language tag). The language tag is listed in the webpage’s or PDF’s properties. This, among other things, is used to help screen readers use the correct pronunciation for words.

10.    Auditors use statistical sampling to select items for audit testing when a population is large and contains similar items. Auditors generally use a statistics software program to choose a random sample when statistical sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

11.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

12.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

13.    The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

14.    A URL uniquely identifies an internet resource, such as a website.

Date published:	May 27, 2025