Guide User Research and Product Testing on Mass.gov

An overview of the research methods used to understand user behavior and test the redesigned Mass.gov.

Real users, real results

At every step of the Mass.gov redesign project, EOTSS Digital Services has sought to incorporate real user feedback into our process in order to tune our design system and ensure that every part of Mass.gov is useful and usable for every user in the Commonwealth.

In order to accomplish this, we’ve taken a comprehensive approach to testing our work with real users, using a variety of testing methods and tools that you'll learn more about below.

Baseline usability testing

80.7 System Usability Score for Mass.gov

In order to understand how users interact with the page layouts, user workflows, and written content on the new Mass.gov, we conducted 11 remote unmoderated usability tests with a total of 97 users during the pilot period. Each of the 11 tests corresponded to a primary state service from one of the topic areas of user interest based on an investigation of website traffic analytics from 2015. These topic areas include unemployment, child support, taxes, business, fishing and hunting, and more.

In these usability tests, users were given a simple task like “You’ve recently been laid off from your job. Show us how you would apply for unemployment benefits in Massachusetts,” and asked to speak out loud as they navigated Mass.gov in order to accomplish the task. While the qualitative, speak-aloud feedback informed our design improvements, users were also asked to rate their experience at the end of each task using the industry-standard System Usability Scale, or SUS, recommended by Usability.gov. According to these ratings, Mass.gov earned a SUS score of 80.7 on a scale of 1-100. The average score for online systems rated using the SUS method is 68.

Information architecture testing

77% Success rate across 10 tasks

As we built out the primary navigation for Mass.gov, we tested the information architecture and main site navigation with 358 users, both online and in-person. Users were asked to complete a series of 10 tasks that involved navigating to key pages using only the primary site menu, and were unable to rely search. Overall, users had a 77% success rate across all 10 tasks, and were able to complete their tests on the first try 70% of the time. However, even for the tests where there was a low success rate, our team was able to analyze the results and make adjustments to the navigation and page content to address some of the key issues

Overview of IA test results

Chart of IA path results

Testing for meaningful accessibility

Ensuring the accessibility of web content to people with disabilities is incredibly important to our team: we firmly believe that Mass.gov doesn’t work if it doesn’t work for everyone. While we’re working with accessibility vendors to ensure that Mass.gov is compliant with state and federal accessibility requirements, we also wanted to go beyond simple compliance and aim for a truly usable experience for the user of assistive technology. Partnering with the Perkins School for the Blind in Watertown, Digital Services’ designers conducted in-person usability sessions with 10 low- and no-vision users. Using a similar method to the baseline usability study, we were able to observe how these users interacted with Mass.gov content while using assistive technology like screen readers. Based on these observations, our team made changes to page layouts and data structures to better support users of assistive technology.

A ZoomText user demonstrates how he navigates a Mass.gov page on a tablet.
A ZoomText user demonstrates how he navigates a Mass.gov page on a tablet.

Join our volunteer test panel

As the official website for the Commonwealth of Massachusetts, Mass.gov is where residents, business owners, and cities find information on services and regulations. We are actively updating our website and seeking volunteers to help evaluate new features and improvements.

Join our test panel for an opportunity to help guide the development of Mass.gov!

Feedback

Tell us what you think