Developing a Test Protocol

Talk Aloud Protocol

One of the best ways to gather information about a user’s experience using your web content is to have them talk aloud as they complete tasks. Record verbal responses as well as any emotional responses or facial expression in a log. You can also ask simple questions like “what were you thinking when…?” to determine their thoughts. It may be preferable to video or audio record sessions and analyze them afterwards so you as the observer can focus your attention on guiding and probing the tester’s thoughts.

Note: If you plan to video or audio record testing sessions, always be sure to inform user testers of this fact.

Test Protocol Example – The Lake Devo Website

As an example of a potential user testing protocol, we will look at the plans put together to user test Lake Devo, a web-based role playing game used by students at Toronto Metropolitan University to design and play out various scenarios related to their subject of study. The Lake Devo website has recently undergone extensive improvements for accessibility.

The User Testers

A couple of screen reader users who had not used Lake Devo previously were recruited to complete a series of tasks that would test the accessibility and usability of key features in the software. Both were skilled JAWS users and knowledgeable about web technology. They were both also students in online courses at the university.

The testers were told they would receive compensation in the form of a $100 CAD prepaid credit card. Regardless of whether the testers completed all the scenarios, they received their compensation.

Pre-Test Tasks

Before starting the testing session, the Observer (in this case the web developer) provided a description of Lake Devo, its purpose, its main features, and an overview of how it might be used to develop a role playing movie. Each user tester was informed at the start of the session that if at anytime they needed a break, felt uncomfortable or wished to discontinue testing, they were free to do so.

Then the Observer introduced the scenarios the tester would be asked to complete and how the session would unfold. For each of the scenarios, the Observer essentially “trained” the user testers by doing the following:

• Reading out the scenario in full
• Walking through the scenario him/herself describing aloud the steps taken to complete the task
• Modelling the talk aloud protocol

Lake Devo Test Scenarios

Scenario 1: Watch the web accessibility movie and answer a skill-testing question

[no login required] low effort

1. Find the Gallery link in the Lake Devo main navigation and open the Gallery.
2. On the Gallery screen enter the words “Lake Devo Accessibility” into the search field and submit the search.
3. In the search results choose the “Lake Devo Accessibility” movie and click or keypress to open the movie.
4. On the movie player screen that opens, find the play button and explore the other options available in the player toolbar.
5. Return to the play button and click or keypress the button to start the movie.
6. Listen to the movie as it plays, and pay attention so you can answer the question about the movie that will follow.
7. Answer the question the Observer asks you.

Scenario 2: Create a new character

[login required] medium effort

1. Log in to Lake Devo with the account created for you, using the login link at the top right of the screen, then fill in the login form, and press the Login button.
2. In the MyStudio screen that opens, find the Access Team community and click or keypress to open the list of movies created by community members (of which you are one).
3. In that list of movies, find the Learning About Accessibility movie, and click or keypress that movie to open it in the script editor.
4. Find the Movie Settings button in the Script Editor, and press it to open the Movie Setting tabpanel, consisting of Movie Info, Character, and Scenes tabs.
5. Click or keypress the Characters tab to open the list of current characters in the movie.
6. In the Create New Character area, open the character editor.
7. Observer describes the character editor: The character editor consists of two main panels, with the preview of the character on the left, and a series of tab panels on the right from which to choose characteristics for the character. These include: info, face, nose, hair, shirt, outerwear, and accessories.
8. To begin creating your character, fill in the name, description, choose male or female, adult or child, and if desired check the checkbox “This character requires a wheelchair.” These are all elements of the form that opens under the info tab.
9. After entering the character info, return to the tabs, navigate to the faces tab and choose a face shape and a face colour.
10. Next open the Nose tab and choose a nose.
11. Optionally, choose hair and a hair colour, a shirt and shirt colour, outerwear, and accessories from the other tabs.
12. After choosing characteristics for your character, press the Save button to save the character.
13. Your character is now created and you will be returned to the list of characters. Find the character you created and click or keypress the Edit link to reopen the character editor.
14. Navigate back to the preview panel of the character editor and listen to the description provided to confirm the characteristics of your character.

You will note that each of the scenarios carefully step the user testers through a specific process on the Lake Devo website. In cases like this one, existing documentation such as the “Help” area of the website often provides a good starting point for writing up the steps involved in a given test scenario.