6 myths that stop companies from doing enough user testing

6 myths that stop companies from doing enough user testing

User testing is not a “nice to have” but a “must have” in software development. Yet many product managers are often so focused on prioritizing features internally that they don’t include user tests in the development cycle.

It’s not like product managers don’t see the value in user testing. We all know the stats around user experience. In fact, in an interview Y Media Labs did recently, many of the product managers we talked to specifically called out user testing as a “must have” requirement for implementing an effective product strategy. Often, though, both product managers and business executives talk the talk but don’t walk the walk.

Personally, I do user testing on every major flow I introduce to my website (full disclosure: I use UserTesting for it). I do it because it’s the right thing to do—and because it saves me from having to do a lot of rework over time. In other words, user testing is an incredibly effective way to avoid technology and user experience debt.

In an ideal world, user testing, as a discipline, should become second nature. But we’re not there yet. Many companies still don’t accept that the cost of not doing user testing is much higher than the time and resources it takes to do the testing and get it right the first time. Below is a list of reasons I’ve seen in my professional career for not doing user testing.

1. We just don’t see the value of user testing

You’d be surprised how many people still don’t see the value in user testing. Of course, most of them would not come out and say it point-blank. One of my favorite excuses is: “Between all of our stakeholders, we have all the institutional knowledge we need to build a great product.” In fact, that is exactly why you need to do user testing.

Many people don’t understand the bias that comes with working day in and day out on a software product. The reality is that internal stakeholders are what you can call superusers. They know the product so well that they are less likely to realize when a new functionality doesn’t quite make sense, or a key feature is difficult for newer users to find.

Internal stakeholders are not your users. You need unbiased feedback from your target market.

Or even if they know something is not quite right, some internal users may simply not speak up for fear of offending their colleagues or embarrassing themselves. That’s the beauty of doing user testing. Your test participants aren’t obligated to spare your feelings. You get the feedback you need, whether you like it or not. Then you can change gears and build a better experience before you waste time on coding, testing, and release efforts.

2. We don’t want to test with people who aren’t our current users

At the end of the day, any testing is good testing. Putting your wireframes in front of people who were not part of the creative process or the requirements-gathering phase is bound to get you some useful feedback.

However, most companies are content with simply showing wireframes to current users. And while you should always test a product with your own existing users, it is also extremely valuable to get some fresh eyes on the user experience you are building or enhancing.

If test participants encounter an issue, you can bet that your own customers will encounter it, too

The nice part about user testing solutions is that they allow you to choose parameters regarding your test participants. You can use filters and screeners that allow you to target the specific group of people you want feedback from. That way, you can test the product with users who are similar to your current customers while still getting a fresh perspective on the functionality you are about to release.

Yet, sometimes, you deal with stakeholders who don’t think that’s enough. No matter how many conditions are listed in the screener, they remain convinced the user test is useless because those users are not their current users. They think the product you’re building is so unique that you absolutely need your current users to test it.

However, with few exceptions, most apps and websites are, in fact, not unique. If you’re working on a retail app, you can certainly get value from tests with users who have made purchases from any number of retail apps similar to yours. Same goes with any other field.

Regardless of whether a test participant is a current user of your app or website, user testing will reveal issues in the overall flow and user experience that will apply to your existing users as well. If you don’t uncover those issues before releasing, you’ll end up getting the exact same feedback from your actual users, after you wasted time and resources and released your feature to production.

3. We don’t have enough internal talent and we can’t hire new resources

If you think you don’t have the budget for research resources, then you definitely can’t afford a bad user experience! It's true that not every department has professional UX researchers or the budget to hire them. But there’s a wide variety of tools that can help anyone learn how to do user testing:

  • A quick Google search for “how to run a user test” can easily help any team member learn the basics of user testing. You won’t become an expert overnight, but I guarantee you’ll be better off than simply dismissing user testing altogether.
  • There are many online training materials available to professionals for nominal costs. I use Udemy every time I need to gain some knowledge on a new topic. For less than $50, you can get 9 hours of video instructions and tips on how to run a usability test.
  • Most vendors offering online user testing will have resources on how to run a test. For example, UserTesting’s free eBook, The complete guide to user testing websites, apps, and prototypes, will help you get up to speed on running user tests even if you have no research background.
  • There are various conferences and seminars throughout the year that you can attend to learn the ropes of user testing and user research.

These days, no company should ever feel like a lack of internal experience should be a deterrent from doing user testing. With so many resources available online, this is a problem with an easy fix.

4. We already know our users/Users are not experts

First, no one really knows “the users” perfectly, no matter how well they think they do. As I mentioned above, internal stakeholders always suffer from a superuser bias.

Second, there’s the “users are not experts” fear. It’s true that users don't always have the technical experience to articulate an issue the way an expert would. Some users give detailed explanations on why they don’t like a specific feature or functionality, some give less detailed descriptions of the issue, and others aren’t sure why they cannot complete a task; they just know something is wrong.

The point is this: users don’t have to be experts. You’re the expert. Not them. You don’t need to get an expert opinion from your test subjects. Instead, what you want to see is whether users have any issues with the flow or functionality you are building. And if they have a problem, it’s your job to determine what the issue is and how to address it.

On a different note, users may not always know what they want, but if they can’t complete a task, they will tell you. Loud and clear. And that’s valuable in and of itself! Sometimes users won’t be able to complete all of the tasks in your test. And that’s a lesson learned.

Sometimes the user will hunt all over the screen and not find what you’re asking them to do. That’s okay too. You’ve learned something. User testing is not about finding expert users; it’s about answering questions: Is what I have built usable? Will the customer use the functionality as expected? If the answer is yes to both, great! If not, go back to the drawing board.

5. We don’t have time for user research

Time to market is a big concern for everyone in software development. But that is no excuse for not running user tests. One of the things I often tell my colleagues is that we’re building software, not doing heart surgery. There is always time for user research if you make it a priority.

Lack of time for user research is directly correlated with aggressive deadlines. As a product manager, you need to balance the project load and release date expectations so that your team can create, execute, analyze, and make changes based on the user test. Once your company accepts the value of user testing, it will just become a line item in the overall project plan. If it takes one more week to deliver the project to production, that’s okay.

There is always time for user research if you make it a priority.

On a different note, you also don’t need to run a user test for every single feature you build. But you need to allocate time for user testing (and making changes based on the feedback) at the requirements phase.

As it happens, this week I was reminded of the power of user testing and the need to allow a user experience architect to act on user testing feedback. Some time back, a stakeholder provided a specific requirement which, in theory, made a lot of sense. We developed the experience in Axure and ran a user test. It performed very poorly; 3 out of 5 test participants were unable to complete the task. We made changes based on the input from the first test, and then we ran it a second time. Unfortunately, even with the changes made, now 4 out of 5 participants either could not complete the task or were incredibly dissatisfied with how the experience was implemented.

The team is back to the drawing board now, and I have informed the stakeholder that the original timeline will be pushed out. A responsible product manager should never release a suboptimal feature that will require rework anyway.

6. We can’t afford user testing, or we’re not sure it’s worth the investment

People who say there’s no ROI to user testing are the ones who don’t understand the cost of bad user experience. Or the cost of software development. Or both. The user experience is something that can very well make or break your website or mobile app.

The usability of a mobile app represents 91% of why a customer will buy from a business through the mobile channel. If your product is clunky, guess what? Some other competitor will do it better, and your user will jump ship for a better user experience. That’s why you have user testing to begin with: to differentiate yourself from your competition by doing something better than others (or, at least, by not doing it worse than your competitors).

I’ve experienced first-hand the cost of shortcuts and bad user experience. In a previous role, we accumulated a lot of UX debt over the first two years of building, releasing, and iterating on a mobile product. We had to do a complete redesign just to bring the app to the point where it was supposed to be, costing millions of dollars.

All of that could have been avoided if we had taken the additional time to do it right the first time. Plus, the cost of running user tests has decreased substantially in recent years especially with the rise in online solutions like UserTesting. The cost of doing several user tests per month is negligible by comparison to all the hours spent on building a feature that people won’t use. Put differently, user testing pays for itself.


User testing, as a discipline, is something that hopefully will become second nature. Most companies obsess about return on investment, customer satisfaction, CSAT and NPS metrics. The beauty of user testing is that it allows you to test your hypotheses way before a feature hits production. It validates or invalidates your designs before you’re measured on the open market, by your customers. To me, incorporating user testing in your project plan is like having medical insurance. Sure, you can choose not do it, but it will cost you big time in the long run!

Originally published at https://www.usertesting.com/ on July 20, 2016.