Audit of the University of Massachusetts Amherst Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the University of Massachusetts Amherst.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the University of Massachusetts (UMass) Amherst for the period July 1, 2022 through June 30, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.  

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Did UMass Amherst’s website and its learning management system (LMS), Blackboard, adhere to Web Content Accessibility Guidelines (WCAG) 2.1 for user accessibility, keyboard accessibility, navigation accessibility, language, erroridentification, and color accessibility?	No; see Findings 1 and 2
Did UMass Amherst ensure that its employees completed cybersecurity awareness training in accordance with Section 1 of Control 14 (Security Awareness and Skills Training) of the Center for Internet Security’s (CIS’s) Critical Security Controls?	No; see Finding 3

To accomplish our audit objectives, we gained an understanding of the aspects of UMass Amherst’s internal control environment relevant to our objectives by reviewing applicable policies and procedures and by interviewing UMass Amherst staff members and management. In addition, to obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Web Accessibility

To determine, for the audit period July 1, 2022 through June 30, 2023, whether UMass Amherst’s website and its LMS, Blackboard, adhered to WCAG 2.1, for user accessibility, keyboard accessibility, navigation accessibility, language, error identification, and color accessibility, we performed accessibility testing procedures on the following:

• a judgmental sample of the 20 most visited webpages during the last month of the audit period, from a population of 25,187 UMass Amherst webpages;
• a random, statistical sample of 60 selected pages with a confidence level⁶ of 95%, expected error rate⁷ of 0%, and a tolerable error rate⁸ of 5%, from a population of the remaining 25,167 UMass Amherst webpages; and
• all 59 Blackboard student features from a population of 59 student features.

User Accessibility

• We determined whether the webpage could be viewed in both portrait and landscape modes.
• We determined whether, when zoomed in to 200%, content on the webpage was undamaged and remained readable.
• We determined whether, when zoomed in to 400%, content on the webpage was undamaged and in a single column.

Keyboard Accessibility

• We determined whether all elements⁹ of the webpage could be navigated using only a keyboard.
• We determined whether any elements on the webpage prevented a user from moving to a different element when using only a keyboard to navigate the webpage.
• We determined whether the first focusable control¹⁰ is a link that redirects to the main content of the website. This is known as either a bypass block¹¹ or a skip link.

Navigation Accessibility

• We determined whether the website contained a title that was relevant to the website content.
• We determined whether there was a search function present to help users locate content.
• We determined whether related hyperlinks allowed navigation to the intended webpages.
• We determined whether headings within websites related to the content of the header’s section.

Language

• We determined whether video content found within the website had all important sound¹² and dialogue captioned.
• We determined whether the language used on the webpage was tagged with the correct language attribute.
• We determined whether words that appeared on the webpage matched the language to which the webpage was set.

Error Identification

• We determined whether mandatory fields on forms alerted users if the field was left blank.
• We determined whether there was a label for any element that required user input.
• We determined whether the label was programmed correctly.
• We determined whether there were examples given to assist the user in correcting mistakes (for example, a warning when entering a letter in a field meant for numbers).

Color Accessibility

• We determined whether there was at least a 3:1 contrast in color and additional visual cues to distinguish hyperlinks, which WCAG recommends for users with colorblindness or other visual impairments.

Cybersecurity Training

To determine whether UMass Amherst employees completed cybersecurity training in accordance with CIS Control 14, we interviewed knowledgeable UMass Amherst staff members and reviewed related documentation.

We used statistical sampling methods for testing, but we did not project the results of our testing to any population.

Data Reliability Assessment

Web Accessibility

To determine the reliability of the site map that we received from UMass Amherst, we interviewed knowledgeable UMass Amherst staff members and checked that variable formats (e.g., dates, unique identifiers, and abbreviations) were accurate. Additionally, we ensured that none of the following issues affected the site map: abbreviation of data fields, missing data (e.g., hidden rows or columns, blank cells, and absent records), and duplicate records. We also ensured that all values in the data set corresponded with expected values.

We selected a random sample of 20 uniform resource locators (URLs)13 from the UMass Amherst site map and traced them to the corresponding webpages on UMass Amherst’s website, checking that each URL and page title matched the information on the UMass Amherst webpage. We also selected a random sample of 20 webpages from UMass Amherst’s website and attempted to trace each URL and page title to the site map to ensure that there was a complete and accurate population of URLs on the site map. We were unable to trace 13 webpages from UMass Amherst’s website to the site map provided by UMass Amherst. We asked UMass Amherst about this and determined that, due to the decentralized administration of UMass Amherst’s website, UMass Amherst management would be unable to provide a site map that listed all the URLs on the UMass Amherst website. See Other Matters 3 below. Although we determined that the URL list provided by UMass Amherst management was not a complete URL list, we proceeded with selecting a sample from the URL list provided and performed the following additional procedure: We received a URL list that listed the 20 most-visited websites during the last month of the audit period. To determine the reliability of that list, we sampled all 20 URLs and traced them to the corresponding webpages on the UMass Amherst website, checking that each URL and page title matched the information on the UMass Amherst website.

LMS Accessibility

As part of our review of UMass Amherst’s Blackboard system, we requested and received access to an online course. To determine the reliability of the Blackboard course we received access to, we interviewed knowledgeable UMass Amherst staff members regarding the student features of the website. Additionally, we used publicly available information from Blackboard’s website to determine what features are available for students and conducted inquiries to determine which features were available to UMass Amherst students during the audit period. We were able to identify 59 features that were available to UMass Amherst students during the audit period. We then traced all 59 features available to UMass Amherst students from the list we obtained from UMass Amherst and from publicly available sources to the Blackboard course to ensure that we received access to a complete and accurate course.

6.   Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

7.   Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

8.   The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

9.   An element is a part of a webpage that contains data, text, or an image.

10.   The first focusable control is the first element a user will be brought to on a webpage when navigating with a keyboard.

11.   This is a link that brings users to the main content of a webpage.

12.   Important sounds are defined as sounds that convey additional meaning or context for the viewer. For example, a horn may indicate a negative sound or a warning, while a bell might indicate something positive.

13.   A URL uniquely identifies an internet resource, such as a website.

 

Date published:	December 30, 2024