Audit of the University of Massachusetts Boston Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the University of Massachusetts Boston.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the University of Massachusetts (UMass) Boston for the period July 1, 2022 through August 31, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.  

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Did UMass Boston’s website andits learning management system(LMS), Blackboard, adhere to Web Content Accessibility Guidelines (WCAG) 2.0 and 2.1, respectively, for user accessibility, keyboard accessibility, navigation accessibility, language, error identification, and color accessibility?	No; see Findings 1 and 2
Did UMassBoston ensure that its employees completed cybersecurity awareness training in accordance with its “Information Security and Awareness Policy”?	No; see Finding 3

To accomplish our audit objectives, we gained an understanding of the aspects of UMass Boston’s internal control environment relevant to our objectives by both reviewing applicable policies and procedures and by interviewing UMass Boston staff members and management. In addition, to obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Web Accessibility

To determine, for the audit period July 1, 2022 through August 31, 2023, whether UMass Boston’s website and its LMS, Blackboard, adhered to WCAG 2.0 and 2.1, respectively, for user accessibility, keyboard accessibility, navigation accessibility, language, error identification, and color accessibility, we performed website accessibility testing procedures on the following:

1. a judgmental sample of the 20 most-visited webpages, from a population of 7,269 UMass Boston webpages. This sample examined these webpages during the last month of the audit period;
2. a random, statistical sample of 60 selected webpages using a 95% confidence level,⁵ a 0% expected error rate,⁶ and a 5% tolerable error rate,⁷ from a population of the remaining 7,249 UMass Boston webpages; and
3. all 59 Blackboard student features from a population of 59 student features.

User Accessibility

• We determined whether content on the webpage was undamaged and remained readable when zoomed in to 200%.
• We determined whether the webpage could be viewed in both portrait and landscape modes (for the 59 student features only).
• We determined whether content on the webpage was undamaged and in a single column (for the 59 student features only) when zoomed in to 400%.

Keyboard Accessibility

• We determined whether all elements⁸ of the webpage could be navigated using only a keyboard.
• We determined whether any elements on the webpage prevented a user from moving to a different element when using only a keyboard to navigate the webpage.
• We determined whether the first focusable control⁹ is a hyperlink that redirects to the main content of the website. The first focusable control is known as either a bypass block¹⁰ or a skip link.

Navigation Accessibility

• We determined whether the website contained a title that was relevant to website content.
• We determined whether there was a search function present to help users locate content.
• We determined whether related hyperlinks allowed navigation to the intended webpage.
• We determined whether headings within websites related to the content of the header’s section.

Language

• We determined whether video content found within the website had all important sounds¹¹ and dialogue captioned.
• We determined whether the language of the webpage was tagged with the correct language attribute.
• We determined whether words that appeared on the webpage matched the language to which the webpage was set.

Error Identification

• We determined whether mandatory form fields alerted users if the field was left blank.
• We determined whether there was a label for elements that required user input.
• We determined whether the label was programmed correctly.
• We determined whether there were examples given to assist the user in correcting mistakes (for example, a warning when entering a letter in a field meant for numbers).

Color Accessibility

• We determined whether there was at least a 3:1 contrast in color and additional visual cues to distinguish hyperlinks, which WCAG recommends for users with colorblindness or other visual impairments.

Cybersecurity Awareness Training

To determine whether UMass Boston’s cybersecurity awareness training met the requirements of its “Information Security and Awareness Policy,” we selected a random, statistical sample of 60 faculty/staff employees out of a population of 3,438 faculty/staff employees, using a 95% confidence level, a 0% expected error rate, and a 5% tolerable error rate, and inspected their cybersecurity awareness training certificates of completion to determine whether they completed the cybersecurity awareness training.  

Additionally, we selected a random, statistical sample of 60 student/graduate employees of out a population of 2,646 student/graduate employees, using a 95% confidence level, a 0% expected error rate, and a 5% tolerable error rate, and inspected their cybersecurity awareness training certificates of completion to determine whether they completed the annual refresher cybersecurity awareness training.

We used statistical sampling methods for testing, but we did not project the results of our testing to any population.

Data Reliability Assessment

Web Accessibility

To determine the reliability of the uniform resource locator (URL)¹² lists that we received from UMass Boston management, we interviewed knowledgeable UMass Boston staff members and checked that certain variable formats (e.g., dates, unique identifiers, and abbreviations) were accurate. Additionally, we ensured that none of the following issues affected the URL lists: abbreviation of data fields, missing data (e.g., hidden rows or columns, blank cells, or absent records), and duplicate records. We also ensured that all values in the dataset corresponded with expected values.  

We selected a random sample of 20 URLs from the UMass Boston URL list that included all UMass Boston webpages and traced each to the corresponding webpage on UMass Boston’s website, checking that each URL and webpage title matched the information on the UMass Boston website. We also selected a random sample of 20 URLs from UMass Boston’s website and traced the URL and webpage title to the URL list that included all UMass Boston webpages to ensure that there was a complete and accurate population of URLs on the URL list.

We also received a URL list that listed the 20 most-visited webpages during the last month of the audit period. To determine the reliability of that list, we sampled all 20 URLs and traced each to the corresponding webpage on UMass Boston website, checking that each URL and webpage title matched the information on the UMass Boston website.

LMS Accessibility

As part of our review of UMass Boston’s Blackboard system, we requested and received access to an online course. To determine the reliability of the Blackboard course we received access to, we interviewed knowledgeable UMass Boston staff members regarding the student features of the course. Additionally, we used Blackboard’s publicly available information to determine what features are available for students and conducted inquiries to determine which features were available to UMass Boston students during the audit period. We were able to identify 59 features that were available to UMass Boston students during the audit period. We then traced all 59 features available to UMass Boston students from the list obtained publicly from Blackboard and identified by UMass Boston staff members to the Blackboard course to ensure that we received access to a complete and accurate course.

Cybersecurity Awareness Training

To determine the reliability of the employee list we received from UMass Boston management, we checked that certain variable formats (e.g., dates, unique identifiers, and abbreviations) were accurate. Additionally, we ensured that none of the following issues affected the list: abbreviation of data fields, missing data (e.g., hidden rows or columns, blank cells, and absent records), and duplicate records. We also ensured that all values in the dataset corresponded with expected values.

We selected a random sample of 20 faculty/staff employees from the employee list and traced their names to the Commonwealth’s Information Warehouse (CIW) to determine whether the list was accurate. We also selected a random sample of 20 faculty/staff employees from CIW and traced their names back to the employee list we received from UMass Boston to ensure that we received a complete and accurate employee list.

To determine the reliability of the student employee and graduate employee list we received from UMass Boston management, we checked that certain variable formats (e.g., dates, unique identifiers, and abbreviations) were accurate. Additionally, we ensured that none of the following issues affected the list: abbreviation of data fields, missing data (e.g., hidden rows or columns, blank cells, and absent records), and duplicate records. We also ensured that all values in the dataset corresponded with expected values.

We selected a random sample of 20 student/graduate employees from the employee list and traced their names to CIW to determine whether the list was accurate. We also selected a random sample of 20 student/graduate employees from CIW and traced their names back to the employee list we received from UMass Boston to ensure that we received a complete and accurate student employee and graduate employee list.

Based on the results of the data reliability assessment procedures described above, we determined that the site map, Blackboard course, and employee list were sufficiently reliable for the purposes of our audit.

5.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

6.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

7.    The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

8.    An element is a part of a webpage that contains data, text, or an image.

9.    The first focusable control is the first element a user will be brought to on a webpage when navigating with a keyboard.

10.    This is a link that brings users to the main content of a webpage.

11.    Important sounds are defined as sounds that convey additional meaning or context for the viewer. For example, a horn may indicate a negative sound or a warning, while a bell might indicate something positive.

12.    A URL uniquely identifies an internet resource, such as a website.

 

Date published:	December 26, 2024