UX research and concept testing: Why polarizing results are ideal

Us designers are encouraged to go wide with our initial design exploration. This is especially important when you’re in the concepting stage: trying out every style, layout, or flow you can think of to ensure you don’t leave any idea unexplored.

Once you complete your exploration, however, you must choose which ideas will go on to be tested. In my last project, I unknowingly created my most polarizing design concept to date—and it reminded me just how important it is to test a wide variety of ideas, including the ones you don’t personally believe are the best solution.

The design challenge

The work was for a health insurance payer. We were exploring how to create a better homepage experience on their customer portal. We had done our research, mapped various user scenarios, identified the pain points, and facilitated a co-creation session with stakeholders. There were a lot of ideas for making this page more of a dynamic dashboard that provides personalized and relevant information that users could act on.

We also wanted to explore a different direction: Rather than looking at how we might create a home page that gives you all the information you need in one place, we experimented with how we would create a home page that helps you navigate the entire portal to find what you need.

At our design review with the stakeholders on the client side, there were strong feelings around Echo. “I don’t like it,” said one stakeholder, followed by another stating, “Actually, this one is my favorite.” The amount of discussion it created made it clear we had to include it in our test.

The research process and what we learned 

We ultimately tested four concepts with eight individuals. With each test participant, we showcased each design concept and asked for initial reactions: what they liked, what they didn’t like, and what they thought was missing.

After reviewing all the concepts, we had the participants rank them from least favorite to favorite. There was a lot of great feedback and insights from each of our concepts, but Echo stood out based on how strongly participants felt about it. Out of the eight participants in our testing, four participants loved it and ranked it as their favorite—while the other four participants hated it and ranked it as their least favorite.

I had never tested a design with such polarizing feedback before, so I was very excited to dig into the details. Going over what we heard from the participants, the insights we pulled from Echo really helped us to understand the different mindsets and approaches individuals have with their health insurance.

Some individuals want to see their personal data and usage so they can better understand the value they are getting out of their insurance (this aligned with our initial hypothesis of what participants would want in a homepage experience). These were the individuals that hated Echo because they felt it didn't provide any context or feedback for how they had been using their insurance—which made it difficult for them to understand the complexities of healthcare insurance and see the value it provided to them.

The individuals that loved Echo had a completely different mindset: Their approach was to get in and out as quickly as possible. Most of their interaction with insurance happens when they are in a heightened emotional state, such as when a big medical bill came through and they were unsure what their insurance would cover. These individuals don’t care about the personalized data; they just want help finding what they need in the complex world that is medical insurance.

Diverse designs lead to better outcomes for users

The insights Echo provided proved invaluable in helping us design a more empathetic homepage experience. This work will remain a highlight of my career for a long time, not only because of how polarizing it was during testing (which is an accomplishment I am immensely proud of) but also because it reminded me of the importance of testing a full range of ideas to truly understand the different mindsets and needs of our users.

It's easy to become attached to our favorite concepts, but we risk missing out on valuable insights if we limit ourselves to only those concepts. Testing a wide range of ideas—even those we may not personally prefer — ensures we gain a wealth of knowledge that enables us to create better products for the people that will use them. Ultimately, that's what it's all about.

Authored by: Conor Flood