Why Remote User Testing Can Be the Best Option to Improve Your Product
User research has been around since the beginning of product design and manufacturing. Back then the prevailing method of testing was in-person user studies. How else would you test a new car without a person driving it in front of you?
Fast-forward 100 years. In the era of internet, remote user testing emerged. Initially it was seen as a cost-effective, “better than nothing” alternative to the in-person testing. However, with the advancement of web development, testing tools and approaches, and with over 4.33 billion active Internet users, remote testing is now a common research method with its own set of advantages.
With the UX market growing annually by 18.2% and potential ROI returns up to 400%, many companies are just getting started with user testing. Remote research may be a great way for them to start acquiring user feedback on their products.
In this article, we'll talk about the key benefits of remote user testing compared to in-person testing. The idea, of course, is not to hard-sell you on remote user research. In-person testing is and always will be a powerful UX research method that UX designers can employ in their process. Debates on "What's better?" will always end with a resounding, "It depends".
Then we will compare two types of remote testing, remote moderated and unmoderated usability studies, so you can choose which method fits your conditions the best.
Advantages of Remote User Testing
Although many companies pick remote user testing over in-person to save time and money, there are benefits that go far beyond being a cheap and quick user research option.
With remote testing you can get insights about how your product performs in different countries, observe users in their natural environment and scale your studies as much as you need.
Let’s talk in detail about what makes remote user testing so efficient.
Cost Effective and Time Effective
No doubt, finding a few participants online is much easier and cheaper than finding or building a lab and then looking for test participants who live in your area. It's even more time-consuming to find test users locally when you require a specific target audience for your product.
With remote testing, you can start performing user studies with just Skype and an internet connection. If you use remote user testing tools, like Teston.io, you won't even have to find participants for your studies — the service will recruit them automatically based on your target audience criteria.
Solution to the Hawthorne Effect
People change their typical behaviour when they know they're being observed. This is called the “Hawthorne effect”, or observer effect, and it was first described back in 1958. This effect is particularly strong in a lab environment.
Imagine yourself eating breakfast at your house. Now imagine eating breakfast in a lab knowing that several people will later be analysing your eating habits. Yes, that's the difference.
The same thing happens when you test your products in an organised research environment. There is a chance that people may act differently. Sometimes it may affect your user research results.
This is where remote studies shine. When you test your product remotely, people usually complete tasks sitting in their home or workplace, i.e., a natural environment.
This is exactly where your real users will be using your products. Thus, the usability insights you gain from the test will be closer to the real world.
Another advantage of remote usability studies is how easy it is to scale them. Even though the most commonly recommended number of participants is five, this number may vary from time to time. Some Google UX designers, for example, had the most success with 20 participants. You'll never know the optimal amount, but during remote studies, you can always recruit new test users more easily than in-person tests. This is true for both moderated and unmoderated usability studies.
Local Recruitment Anywhere
With remote user testing you can easily recruit participants from different parts of the world. You won't have to travel every time you need insights from a new foreign market. This is especially useful for SaaS and B2C companies with services that operate globally and have users from multiple countries.
It's important to understand that while key usability guidelines hold true worldwide, user behaviour can vary significantly in different countries, and remote user testing can bring to light the usability issues in diverse environments.
If you decide to perform remote usability research for your product, there are two options available: moderated user studies and unmoderated user studies.
During moderated user testing, there is a researcher or a facilitator who guides test users through the tasks. In unmoderated research, users perform tasks at their own pace with no facilitator present.
Both moderated and unmoderated user research have the advantages listed above, yet they have key differences as well. Whether you pick moderated testing or unmoderated testing depends on a number of factors. Let's take a closer look at each one and learn how to pick the right method for your project.
Moderated Usability Testing
Put simply, moderated usability testing is when you or a UX researcher is present while users complete tasks with your product. In a remote scenario, you watch them performing tasks via Skype or any other tool that enables screen sharing in real-time. This type of testing enables you to ask users follow-up questions and is generally less structured than an unmoderated test session.
Moderated usability testing is suitable for both validating your concepts and getting qualitative insights about your product. Below are the key benefits of moderated testing sessions.
Back and Forth Dialogue With Participants
When you have the ability to watch how people interact with your product in real-time, you can notice things that unmoderated, automated tests simply can't convey. For example, when people use your low-fidelity prototype or even wireframes, their very first reaction can be asking you, "What is this?" In an automated study, they may simply leave or proceed with the tasks, and perhaps the initial confusion would fade off, but this critical insight about their user experience maybe missed.
Although UX researchers often have a script with tasks and questions they'd like to address during the study, sometimes things don’t go as planned. This isn’t a bad thing because human behaviour with all its complexity can never be 100% predicted. At the same time, you get an opportunity to test and observe behaviours you'd never think of during your preparation. If the users misunderstand the questions in your script, you also have an opportunity to elaborate and clarify what you mean. This flexibility is an additional benefit.
While there are lots of advantages, there are also drawbacks to moderated studies.
The Observer Effect Is Still Present
Even though participants perform tasks in their natural environment, they still may modify their behaviour due to the presence of a UX researcher. For example, they may not like the product or may deem it inconvenient, yet never voice their concerns out of politeness.
An experienced UX researcher should be aware of the observer effect and mitigate it as much as possible.
Recruiting for moderated remote studies takes a lot of time. You have to find potential candidates and then filter them. If you’re conducting the user testing sessions yourself, you also have to be present all the time. Otherwise you risk losing the key benefits of moderated user testing, such as close interaction with participants and the ability to ask follow-up questions. An average testing session lasts around 30 minutes, so make sure to allocate the appropriate amount of time in your schedule.
Unmoderated Usability Testing
Remote unmoderated usability testing is conducted with no observer or facilitator present. Users are free to perform tasks at their own pace and in their own environments. You are then presented with either a video recording of their results or raw text data.
The accuracy of the results in this type of testing is highly dependent on the clarity of a test script. As unmoderated user testing is typically conducted through specific user testing services, the quality of studies also highly depends on the quality of the screening process for potential test users.
Below are the key advantages of unmoderated usability testing.
Unmoderated usability testing is easy to set up, especially with devoted user testing services that help find recruits from the target audience, as well as providing support when you create and edit your testing script with questions and tasks.
With unmoderated tests you can test your interface elements with many more participants, getting qualitative data insights with quantitative backing. This allows you to more confidently generalise your findings compared to testing with a smaller number of participants.
Less Time Consuming
Although the time you spend on preparing for moderated and unmoderated research is comparable, you don’t have to be present during unmoderated user testing. Also, unmoderated user testing services do all the recruitment for you. They help to find quality candidates, and you don’t have to spend time on finding replacements if someone doesn't show up for the testing session.
Of course, unmoderated user research also comes with its own drawbacks that you should take into account.
No Back-and-Forth Dialogue With Participants
At the end of the study usually you get a video or data about the participant's session. You're unable to ask any follow-up questions because you'll be watching a recording.
The Number of Useful Insights Depends on the Quality of Your Research Questions
You need to spend time ensuring there are no ambiguous tasks in your study. Otherwise, they could be misinterpreted. If the users gravely misunderstand your questions and tasks, the results will be moot.
Beware of Certain Types of Recruits
Some people take a lot of unmoderated tests and participate in unmoderated research tests to make quick money. They may not always be interested in providing valuable information or demonstrating genuine behaviour with your products. Sometimes they will say the things they think will please you most so you’re happy with their testing result. Of course, the whole point of user testing is to improve your designs, but they don’t always understand that.
Make sure the user testing service you're using takes the recruiting aspect seriously by filtering out super testers, ensuring participants are honest and encouraging them to be outspoken about your product when they don’t like it.
How to Pick the Right Method for Your Remote Studies
In a perfect scenario, you will use both unmoderated or moderated user studies, taking advantage of the best aspects of each.
If you need to quickly check whether your Android mobile app or prototype is user-friendly without spending a week looking for users from the target audience, go for unmoderated usability testing. The same applies if you need to validate an unambiguous and specific feature — for example, a login page for your mobile app.
If you need to gain deep insights about user experience or validate early design concepts, go for moderated usability testing. Then you can create a more robust product prototype and test it on a larger number of participants in an automated, unmoderated study later on.
Now that you know what makes each user testing method so powerful, it’s time to apply your knowledge. It's up to you whether you decide to conduct moderated user testing yourself or use tools for unmoderated user research. One thing is clear: Your product will soon become much better.