Introduction to Usability Testing

Every product or service in the world has users and each user can have their own way of utilizing what they have been given. User testing is the process where a product or service is put through the paces to see if it is actually usable, what parts may be working like a dream, or where there needs to be improvements. User testing is typically performed on websites, paper prototypes, high fidelity prototypes, applications in development, or “completed” websites and applications.

Usability Test Methods

There are a large number of tests that can be performed with users, all of which documentation can be found online in a number of places. The type of test method depends on:

  • the level of feedback needed
  • the amount of time allotted for user testing
  • what is being tested

The most important thing to remember is to figure out those three items and then pick which method would best suit those criteria.

In this section, we’ll be going through the types of user tests that I find to be most useful. Each test is designed to get different types of feedback, depending on the goals of what is being tested. This should help you decide which test is right for you at any given time.

 

Asynchronous

Closely related to the RAW method, this is a relatively new testing method that can return varied results. An asynchronous user test is one where the tasks are laid out in some type of survey and users are asked to go through them at their own pace. What makes this...

Card Sorting

Card sorting is a method that is typically used when testing navigation and information architecture. This is a good method to use when beginning a project to get an understanding of how users expect to use your app or website. When beginning a card sorting session,...

Cognitive Walkthrough

A cognitive walkthrough is a great method to do before performing another type of user test. In my experience, this is good to do before a Flash or RAW test because it allows us to be able to get to the heart of what we want to know, what should be tested, and can...

Flash Test

Flash tests are the quickest test method and are meant to be done as rapid tests. The goals of flash tests are to get information from as many participants as you can within a set amount of time. A single flash test should be no more than 5 minutes long when giving...

Focus Group

A focus group is good for evaluating a section of the overall product or service. Creating a focus group can take a number of different approaches, whether it is having a small group of users discuss what you want feedback on or whether you combine the focus group...

RAW (Real Actions Witnessed)

RAW user tests are a time intensive testing method. With each RAW test, stakeholders are brought in to watch a group of participants use a product or service in real-time. Typically users will be in one room performing the test and stakeholders will be in a separate...

Survey

Surveys are good for receiving feedback if you are simply asking questions and not having users do any tasks. These are always a numbers a game because you typically will not receive responses from all that are sent. These can be created with any number of online...

Synchronous Video

The Synchronous video usability testing method is one that can be combined with other methods such as Cognitive Walkthrough, Flash, or RAW tests. This is a great method when you want to be able to perform a usability test with remote participants but you are looking...

 

Recording Results

Results can be recorded in a number of different ways, but they must be recorded somehow to be able to report on them to stakeholders. This step is important to be able to show the data to validate future design decisions and to show how users are actually using what you are testing.

When performing usability testing, make sure that your institution has clearance with their Institutional Review Board (IRB). This is important when doing usability testing with human subjects. You can find more information about IRBs on this FDA page of frequently asked questions. If you are performing usability testing in the European Union, make sure to comply with GDPR rules in regards to collection personally identifiable information. A good article can be found about GDPR and user research on Medium.

 

Note Taking

Taking notes is an essential part of user testing and each user testing method can have a different approach to notes. When taking notes for a usability test, it is best to have a dedicated person whose only job within the test setting is to take notes. This will...

Survey Collection

Collecting a survey is another method of taking notes where the control is relinquished to the users and what they are willing to write and submit in a survey. These surveys can be created with Qualtrics, Google Forms, or other survey products, as long as they can be...

Video And Survey

Some user testing methods allow for recording results to be done with a combination of video, surveys, or other note taking. The RAW and Asynchronous methods can involve video where the user’s actions, and sometimes anything they say, are recorded. These would need to...

Reporting

Creating a report from all of the user testing results is, arguably, the most important part of the test itself because it allows you to show others what the test found to be true. There are few aspects of tests that should be included in a report to show the most from the results and help inform any decisions that the report will create. Everyone who views the report will likely look for something different from it, whether that is a stakeholder wanting just a summary or a designer who will be looking at specific user actions.

Here is an example of a full report for a Flash test.

1. Summary

The summary of a report is usually the first page so that anyone reading can get a succinct overview of the user test and its results. This page should include any prominent feedback such as the number of users who did a certain action or if there is something that the test found that it wasn’t actually testing. Try keeping the summary to a single page, include the name and date of user testing, and a summarized version of the recorded results.

A key piece of information that should be included in the summary are the stumbling blocks that users encountered during the usability test. This will allow stakeholders to know what should be prioritized.

 

2. Demographic Information

Not all user tests need to collect demographic information but if it is collected, it should not be personally identifiable such as a student’s school ID number or name. This type of information is important for making sure that you are testing with the target audience or so that you can see who is actually using an application or product (to potentially adjust the audience in the future). Some of the information that is good to collect would be a student’s year and their major or area of study. Depending on what would be relevant to the end results, collecting more information could be helpful as long as it is not personally identifiable.

 

3. Recorded notes/survey results

This section of the report is where the compiled results will be added. If a survey was used, add in all of the responses and any additional notes that were added. Any notes that were taken outside of a survey should also be added in here. This section will be the longest, but it can be combined with the goals and tasks instead of adding them separately. Not everyone who receives this report will want to look at these, since they are raw notes taken, thoughts recorded, and actions observed. This section is important for making decisions on how to make any changes to what was tested from a designer or developer point of view because it helps them to know as much of the data and process as possible.

 

4. Reflection of goals and tasks

Reflecting on the goals of the user testing and what tasks were asked of the users is important to be able to make sure that the results of the test matched what was expected to be learned. If the goals and tasks do not match the results, there may be a need for further user testing and that is just as important to know as a test that validates the thoughts going into a test. This can be combined with the recorded notes/survey results when breaking a report down by task and listing each note under a task or goal heading.

Usability Test Process​

Creating a usability test and performing it comes with a quite a steps to be done properly. A test should always be created based on goals focused around what you want to learn from that particular test. Once you have the goals and the area of an application or website you are testing, you can determine the audience that should be tested, who will typically use this feature.  Then you will create tasks for the user to perform, which will help determine the type of user test that should be used.

Once you know the type of test, you can begin performing the user test and then report the results back to others. It is always good to think of each user test as a stepping stone to lead to future testing and create recommendations based on that. Continue reading below for each of these steps in-depth.

1. What are you testing?​

The very first step in a usability test if determining what it is that you are going to test. This will impact every subsequent section and can be iterative if there are multiple items to be tested. If you aren’t sure exactly what you want to test from a larger application, the first step is to narrow down to a couple of actionable items that users can perform. From there, you will be able to go through the process and narrow the scope to create a single usability test. Remember that usability testing shouldn’t try to capture large, complicated items all at once but should instead focus on breaking those down into core elements that can be tested individually to create a more complete picture. 

2. Determining goals

Before performing a user test or even creating tasks, you should figure out what you want to learn from the test. What are the goals of a certain feature that you are looking to test? Is this feature meant to allow users to get to a different part of the website or application, or is it a way for them to submit feedback? Knowing what you want out of a user test will help you to determine who should be recruited for the test and what tasks the user may be expected to perform.

What do you want this test to achieve? Each test should have a narrow scope of a feature or two, depending on how complex, and the goals should be focused on the future of the feature. Is this something that is currently in development and you need to take the feedback to developers? Is this a feature that has been fully developed and needs to be seen if users can actually use it? Knowing who will be receiving a report from a particular user test can help to determine what you want to learn from the test.

3. Recruit team and assign roles

Most, if not all, usability tests require more than one person to run effectively. The roles needed will depend on the method chosen and may differ slightly across tests. You will want to recruit your team and assign the roles in advance of performing any usability test so that each individual has an understanding of what is expected of them.

There are a few roles that you will have in most usability tests regardless of method.  A facilitator is responsible for engaging with the participant and helps to run the test. A note taker is responsible for recording the results and capturing the answers to any test questions. A recruiter is a key individual who will be responsible for recruiting participants for the usability test, whether it is done ahead of time or, in some cases such as a Flash Test, on the fly.

4. Audience

Once the goals have been determined for your user test, the next step is figuring out the audience who you want to test this feature. Is this something that a student worker will use when checking out a book to a patron? Will staff also use this feature in the same manner? Audiences tend to vary from test to test and knowing the audience will help determine what tasks should be developed and what type of test method will be appropriate.

A user test may have more than one target audience who can and should be involved in testing a feature. In this case, it would be best to determine whether any tasks in the workflow are different between the audience types or if they are the same. If these tasks differ, it may be best to do two user tests, one with each audience, to get a more well-rounded view of the feature and how users will interact with it.

Some tests have audiences that will use other portions of an application or website but not the feature that is currently on the table to be tested. In this case, are there any groups that should not be included in the user test?

5. Developing user tasks

User tasks are the cornerstone of a user test, as without tasks there is nothing to test. Tasks should be developed based on the audience and goals that have already been established.

When creating tasks, a user should not be led by the question to do a certain action. These can be done either as questions or directions that the user should complete. Let’s combine our goals from above and create some user tasks for them.

Goal: Understand where the user expects to find information on our loan policies.
Don’t Say: “Click on Services and go to Policies.”
Do Say: “Where would you expect to find information on your loan fees?”

Goal: Do student workers understand how to find a patron by name?
Don’t Say: “Look up the patron in the Check out section and search for the patron by name.”
Do Say: “How would you find a patron who does not have their ID card?”

These examples show a task that does not lead the user and one that does. Keeping to non-leading tasks will create more natural feedback from a user test and allow us to see how a user expects to work with our feature.

It can be difficult to create non-leading tasks when there are specifics that need to be completed before the task itself. In this case, you should start the user on the page with those already in place and explain that as part of the beginning of the test. A good way to know if your tasks are understandable and not leading is to do a Pre-Test Dry Run, as defined in the methods above. This can be done with a group of people that are not necessarily your target audience, as you are not trying to understand the result of their actions but rather if they are understanding what they need to do. In a dry run, it is common to reword tasks so that they are getting at the heart of the goal and won’t lead the user to the conclusion you are hoping for from the test.

When you have the tasks developed, there are few more things that you should think of in regards to those tasks before moving on to determining the type of user test:

  • Where will users end up at the end of one task and where should they begin the next? Sometimes you will need to instruct users to go back to a different page before starting the following task.
  • If you have multiple users that need to type something into a search box or log in to the same system, are you going to clear the cache between each test so that users start fresh?

6. Determining type of test

Once you know the user tasks that you wish to use for a test, you can figure out what testing method would be best. There are a couple of factors that are involved in this decision: location, amount of time expected of users, amount of time expected of test givers, and amount of information that needs to be gathered. See our previous section on Usability Testing Methods to read more about the various methods collected in this toolkit.

Let’s create a few scenarios and determine the best test method.

Scenario One
Goal: Understand where the user expects to find information on our loan policies.
Task: Where would you expect to find information on your loan fees?
Information: User thought of organization of features and ability to navigate, potentially organization of content
Location: Anywhere with access to application

Scenario One can be complicated because there is a lot of information being packed into a single task and user test. Depending on the level of depth needed in the collected information, this could be done with a Flash test, Cognitive Walkthrough, or Card Sorting. If you are looking purely for content organization and do not have much time for the test, card sorting or a flash test would be sufficient. If the information is more on understanding the navigation of the feature and how users would think about that organization, a cognitive walkthrough would be ideal because it allows for deeper discussion on features and can even result in recommendations for the report. The cognitive walkthrough would require more time for both users and test givers but it will result in more qualitative information and could lead to how to perform another test in the future.

Scenario Two
Goal: Do student workers understand how to find a patron by name?
Task: How would you find a patron who does not have their ID card?
Information: Limited to navigation for this task
Location: Anywhere with access to application

Scenario Two could be done in a number of methods, including a RAW test, Flash test, or Asynchronous. The choice from here becomes the amount of time able to be given by the test giver. If not much time can be allocated and you are trying to get more quantitative data on the number of student workers that understand it versus do not understand, you could do a Flash test. If more time can be done, a RAW test could be completed of watching a user interact with it and follow up with a more structured testing session. Asynchronous could be accomplished if there is no time up front for a there is a good amount of time that can be dedicated after results have been compiled, as video and survey results would need to be read and evaluated.

7. Pre-Test Dry Run

This step can come before determining the type of test or after if you know what test method you think would be best. Performing a dry run is a crucial step in performing a usability test because it will allow you to refine user tasks and find the most appropriate type of test method to use for the results you are seeking. This step can be iterative and performed multiple times.

A dry run should function as similarly to an actual usability test as possible. The team that was assembled in step 3 will be the ones performing the dry run. This is a chance to go through all of the user tasks, understand what is being tested, and ask any questions that may come up surrounding the tasks or functionality of what is being tested.

8. Recruiting usability testers

No usability test can be performed without testers to participate. These users should be pulled from your audience that was determined in step 4. This will always depend on the method of test you are planning to perform. If you are doing a Flash test, users are typically recruited on the spot at the test but if you are doing a Focus Group, you will need to have these users recruited in advance. Please refer to each individual method for the recommended number of users and when it is best to recruit them.

9. During the test

A usability test should begin in the same way for each user. It is important to tell the user what you are testing, that you are testing the system and not them, and to let them know any specific details about the test. These specifics could be anything, such as if you need to have a permission form signed or if they are going to receive compensation at the end. Make sure to check the guidelines for your Institutional Review Board (IRB) or GDPR information to see if you are required to have users sign a form before participating in your usability test. If you are collecting demographic information, it is good to let the user know that it is being collected and that it’s nothing that will personally identify them. 
When performing a user test, it is helpful to have multiple people present, each with a different role.
  • You will have the note taker, who will be taking all notes from the test.
  • The facilitator is the one who speaks with the user, tells the tasks and answers any questions or prompts for more feedback.
  • Depending on the type of test, you may have a recruiter who is in charge of getting participants. This is most typical in a flash test setting.
Users will often have questions during a test and as the test giver, we don’t want to actually give them directions unless they are truly stuck on a task. The best way to handle user questions is to ask a follow up question of them that will prompt the user to talk about what they are thinking or what they are trying to do. For example, if a user asks “What does this button do?” you could ask “What do you think that does?” or “Where would you expect that link to take you?” Asking these questions will help to get more information from the users and understand how they are thinking they should be able to work within your product.

 

If a user is struggling and can not continue through the task, that is also good feedback from a user testing standpoint. It is a good first thing to ask if a user is struggling is “What are you trying to do?” or “What are you looking for on this page?” These questions should help you as the test giver to understand how the user is thinking and possibly give them a hint of where to look, without doing it directly. If users continue to struggle, this is a time when it is okay to tell the user what you were expecting, but should always make a note that the user could not complete the task and had to show them.

 

Now that you’ve answered user questions and moved through the task(s), how do you know when you’ve collected enough information? You should always ask yourself if you’ve answered these questions:
  • Have you answered the goal of the task and has your user completed the task?
  • If your user hasn’t completed the task, do you know why they couldn’t?
  • Have any recommendations from the user been taken as notes?
If all of those questions are a yes, then you’ve likely collected enough information and can conclude the test with this user. Making sure that all of these are completed is more simple when you have all of the tasks written where you are taking notes so that you can follow along and make sure that they are updated at each step of the user test.

10. After the test

When the test is completed, there are a few things that should be done to indicate that the test is finished for the user. You should always thank the user for their participation and for helping to provide feedback on the product that you were testing. The note taker should be sure that all data is recorded in the appropriate location. If the test comes with compensation (money, a gift card, candy, etc.) then that should be given to the user after thanking them and making sure all of the data is saved.

 

In the case of asynchronous testing or surveys, these will likely consist of a thank you page with any relevant information and making sure that any video software that was used has been set up to save the recording. Make sure that someone is alerted when something is saved in a survey or asynchronous test so that any issues can be found early, if present.

11. Reporting back to others

After creating the report from the test results, it is typical to report these results back to stakeholders. When going over the report, it is best to focus on the summary and any standout cases where users excelled or struggled. These results should be reflective of what data was recorded and not play into any politics or favoritism for a specific result from the stakeholders. Bringing the most important results that are actionable to the summary is a good way to deliver the most relevant information to the most parties at the beginning of the report.

 

When completing the report, it is important to know where it should be sent and what information is needed. The full report may not need to go to all stakeholders and this is important to know beforehand. Are there action items that should be put into some form of issue tracking software instead of being sent by email? These logistic elements of the workflow will help to keep the feedback loop concise and make sure that all appropriate parties are receiving the correct information.

 

Always leave room for questions from stakeholders about the results. If they are not present at the test session, they will not have that first-hand experience to be able to understand exactly what the user was doing and may require additional information that was not previously thought about.

12. Using results as a stepping stone for future testing

All user tests should be considered as a stepping stone for future user testing. Whether it is from the viewpoint of validating a decision or giving areas for improvement within a feature. Did any test results give positive results with nothing that needed to be adjusted? That portion of the feature can be validated as usable by target users and set aside for now, as it should be tested again in the future as features continue to change.

 

Did any results give feedback where the tests were inconclusive, such as half of users understanding a feature and half not? This should lead to performing another user test and perhaps breaking down a task or separating a goal into multiple pieces. Sometimes user tests will be inconclusive and the best thing to do is re-evaluate the goals, tasks, and test method before doing another user test.

 

No matter what the results reflect, there is always more user testing that should be done with other features or more development and design thought that should be applied. Testing a feature now and having positive results does not mean that the feature should never be tested again, as users and technology changes, so any feature can be revisited and improved as time goes on.

Glossary

There are a number of terms used within this documentation that may be unknown to you. Please use this glossary to help get a better understanding of these terms.

 

Note taker – Team member tasked with taking notes during the usability test.

Facilitator – Team member tasked with speaking with the participant and asking users to perform tasks.

Recruiter – Team member tasked with recruiting participants to take part in the usability test.

Participant/tester – User who will be participating in the usability test.

Institutional Review Board (IRB) – An administrative body established to protect the rights and welfare of human research subjects recruited to participate in research activities conducted under the auspices of the institution with which it is affiliated

GDPR – The EU General Data Protection Regulations (https://eugdpr.org/)