Audit of the Appeals Court Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Appeals Court.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor (OSA) has conducted a performance audit of certain activities of the Appeals Court for the period July 1, 2023 through June 30, 2024.

We conducted this performance audit in accordance with generally accepted government auditing standards,¹⁰ except for the following areas:¹¹

• Paragraph 8.39 of Chapter 8 (which pertains to determining whether internal controls are significant to the audit objectives);
• Paragraphs 8.59–8.67 of Chapter 8 (which pertains to information systems controls considerations);
• Paragraphs 8.72–8.76 of Chapter 8 (which pertains to fraud inquiries);
• Paragraphs 8.77 and 8.78 of Chapter 8 (which pertains to identifying sources of evidence and the amount and types of required evidence);
• Paragraph 8.80 of Chapter 8 (which pertains to determining whether other auditors have conducted, or are conducting, audits that could be relevant to the current audit objectives);
• Paragraph 8.90 of Chapter 8 (which pertains to obtaining sufficient, appropriate evidence to provide a reasonable basis for addressing the audit objectives and supporting their findings and conclusions); and
• Paragraph 9.50 of Chapter 9 (which pertains to obtaining and reporting the views of responsible officials from the auditee concerning the findings, conclusions, and recommendations in the audit report, as well as any planned corrective actions).

We believe that, except for the information outlined in the “Scope Limitations” section, the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is our audit objective, indicating the question we intended our audit to answer; the conclusion we reached regarding our objective; and, if applicable, where our objective is discussed in the audit findings.

Objective	Conclusion
Did the Appeals Court’s website adhere to the World Wide Web Consortium’s (W3C’s) Web Content Accessibility Guidelines (WCAG) 2.1 for user accessibility, keyboard accessibility, navigation accessibility, language accessibility, error identification, and color accessibility?	No; see Finding 1

Scope Limitations

Paragraph 9.12 of the US Government Accountability Office’s Government Auditing Standards states, “Auditors should . . . report any significant constraints imposed on the audit approach by information limitations or scope impairments.” While performing the audit, we encountered the scope limitations described below.

OSA has completed performance audits of the Appeals Court in the past. Two such audits were released in 2015 and 2020. The 2015 Appeals Court audit reviewed “whether . . . the Appeals Court’s case-management system was operating effectively” and whether “appeals were managed, heard, and decided in a timely manner,” and whether “appeal entry fees were properly waived in accordance with applicable requirements.” The 2020 Appeals Court audit “examined the Appeals Court’s compliance with Section 4 of Chapter 262 of the General Laws and Rules 8a–8f of the Massachusetts Rules of Electronic Filing” as well as “the efficiency of the electronic document filing system eFileMA,” and the 2020 audit also “assessed the [Appeals Court]’s fee collection.”

In response to OSA’s letter of intent to conduct a performance audit (see Appendix A), the chief justice of the Appeals Court formally declined participation via written correspondence (see Appendix B). The chief justice asserted that the audit would exceed OSA’s statutory authority under Section 12 of Chapter 11 of the General Laws.

The chief justice further contended that WCAG 2.1, established by W3C and required by EOTSS, are not government regulations and, therefore, are not enforceable against the Appeals Court. The chief justice maintained that OSA does not have authority to require compliance with, or to audit against, WCAG 2.1. Instead, the Appeals Court stated that it follows the requirements of the US Department of Justice’s Final Rule, Section 35 of Title 28 of the Code of Federal Regulations (Civil Rights Division Docket No. 144, 89 Final Rule 31320),¹² which uses Level AA of WCAG 2.1 as the technical standard for accessibility under Title II of the Americans with Disabilities Act. The deadline established by the US Department of Justice to ensure that websites are compliant with Levels A and AA of WCAG 2.1 is April 24, 2026.

Additionally, while the Enterprise Information Technology Accessibility Policy, issued by the Executive Office of Technology Services and Security (EOTSS) in 2021, requires agencies using EOTSS services, such as Mass.gov, to comply with Levels A and AA of WCAG 2.1, the chief justice of the Appeals Court argued that this policy does not create a legal obligation for the Appeals Court to follow WCAG 2.1 and reaffirmed that OSA does not have enforcement authority in this regard.

Consequently, the Appeals Court declined to allow us to conduct our audit, even though OSA made efforts to resolve the jurisdictional issue by holding meetings with Appeals Court representatives to clarify this issue. As a result, we were denied access to essential information required to accomplish part of our audit objective.

As a result of the Appeals Court’s denial to grant us access to essential information required to accomplish our audit objective, we independently compiled a list of URLs¹³ from the Appeals Court’s Mass.gov website¹⁴ in order to test our objective. When selecting URLs, we relied on our professional judgement¹⁵ and the webpage tagging system within Mass.gov. The list of URLs obtained through this approach may not be a complete population of the Appeals Court’s webpages.

Web Accessibility

To determine whether the Appeals Court’s website adhered to the W3C’s WCAG 2.1 for user accessibility, keyboard accessibility, navigation accessibility, language accessibility, error identification, and color accessibility, we took the actions described below. First, we performed accessibility testing procedures on a random, nonstatistical¹⁶ sample of 60 Appeal Court webpages out of a population of 214. We performed the procedures described below on the sampled webpages.

User Accessibility

• We determined whether content on each website could be viewed in both portrait and landscape modes.
• We determined whether content on each webpage was undamaged and remained readable when zoomed in to both 200% and 400%.

Keyboard Accessibility

• We determined whether all elements¹⁷ of each webpage could be navigated using only keyboard commands.
• We determined whether any elements on each webpage prevented a user from moving to a different element when using only keyboard commands to navigate the webpage in question.
• We determined whether the first focusable control¹⁸ on each webpage was a hyperlink that would redirect users to the main content of the webpage.

Navigation Accessibility

• We determined whether each webpage contained a title that was relevant to the webpage’s content.
• We determined whether there was a search function present to help users locate content.
• We determined whether hyperlinks correctly navigated to the intended webpages.
• We determined whether headings within webpages related to the content of the header’s section.

Language Accessibility

• We determined whether any video content found within each webpage had all important sounds and dialogue captioned.
• We determined whether the words that appeared on each webpage matched the language attribute¹⁹ to which the webpage in question was set.
• We determined whether any webpage sections that contained language differing from that to which the webpage was set contained their own specified language attribute.

Error Identification

• We determined whether mandatory form fields alerted users if they left these fields blank.
• We determined, for form fields that required a limited set of input values, whether users were alerted if invalid values were entered into these types of fields.
• We determined whether there were labels for any elements that required user input.
  • We also determined whether these labels were programmed correctly.
• We determined whether examples were presented to assist users in correcting mistakes (for example, a warning when entering a letter in a field meant for numbers).

Color Accessibility

• We determined whether there was at least a 3:1 contrast in color and additional visual cues to distinguish hyperlinks, which WCAG recommends for users with colorblindness or other visual impairments.

We used nonstatistical sampling methods for testing and therefore did not project the results of our testing to any corresponding populations.

For this objective, we found certain issues during our testing; see Finding 1 for more information.

The Appeals Court’s denial to grant us access to essential information materially affected our ability to carry out all necessary procedures in the bulleted list above, including a data reliability assessment. Therefore, in accordance with generally accepted government auditing standards, this report discloses a scope limitation.

10.    Generally accepted government auditing standards is an auditing term used to refer to the standards issued by the US Government Accountability Office, which are listed in its Government Auditing Standards. (The version referenced in this report is the April 2021 technical update, GAO-21-368G.) These standards provide a framework for conducting high-quality audits of government programs and entities. They outline the ethical principles, professional qualifications, and performance requirements that auditors must follow when conducting financial and performance audits—as well as other types of engagements—to ensure accountability and transparency.

11.    We were unable to satisfy the requirements of the listed generally accepted government auditing standards because the auditee refused to participate in the audit.

12.    You can find the text of this rule here.

13.    A URL uniquely identifies an internet resource, such as a website.

14.   Webpages within Mass.gov can be tagged with the names of Commonwealth entities, thus indicating which Commonwealth entity authored the webpage. In the case of the Appeals Court, a user would see something like “Offered by Appeals Court” underneath the website header and above the webpage heading. See the “Accessibility Features of a Website” image in this report for an example of this.

15.  Professional judgement, in this situation, involved auditors reviewing the content of the sampled webpages, as well as the path to navigate to said webpage. For example, the webpage “Appeals Court Resource Flyer 7-17.pdf | Mass.gov” (see https://www.mass.gov/doc/resources-flyer-for-self-represented-pro-se-litigants) is a webpage that auditors could navigate to using only hyperlinks within webpages tagged with “Offered by Appeals Court.”

16.    Auditors use nonstatistical sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review.

17.   An element is a part of a webpage that contains data, text, or an image.

18.   The first focusable control is the first element a user will be brought to on a webpage when navigating with a keyboard. If first focusable control also redirects users to the main content of a webpage, then it is known as a bypass block or a skip link.

19.    A language attribute (also known as a language tag) identifies the native language of the content on the webpage or PDF (e.g., a webpage in English should have an EN language attribute). The language attribute is listed in the webpage’s or PDF’s properties. This, among other things, is used to help screen readers use the correct pronunciation for words.

Date published:	August 6, 2025