The user testing mistake you must avoid


Justin Ternullo

Justin Ternullo

Blog author Justin Ternullo

Justin Ternullo is the Chief Design Officer at PolicyGenius, a digital insurance brokerage trying to make sense of insurance for consumers.

Published July 30, 2014|4 min read

Policygenius content follows strict guidelines for editorial accuracy and integrity. Learn about our

editorial standards

and how we make money.

News article image

Like any startup trying to craft a product that solves a real problem, we're always user-testing. Whether it's a completely nascent idea or the development of a new feature, we're constantly trying to get feedback from our actual users. You could call this an 'ostrich test' - referring to the act of removing one's head from a hole (or anywhere else...) to see what's really going on. We want to make sure that we're not building in an echo chamber. Real, unfiltered, unbiased user feedback is the juice that keeps us pushing the product further, and tailoring the experience to be easy, quick, useful, and hopefully fun.We do a lot of in-person user tests, particularly during the early stages of a project. We invite a group of people to meet with us, usually with little to no idea what PolicyGenius actually does, for a one-on-one feedback session. The sessions themselves range from 15 minutes to over an hour. And part of our user testing 101 is to start with some little white lies.It begins right-away with the introduction. I'll sit down with the user to begin the test, and with a slightly crumpled brow say: " Hi, I'm Justin - and I DID NOT design what you're about to see. In fact, I don't work for PolicyGenius at all. I'm an independent researcher hired to get your insight on a few ideas they have."

No blinking. No winking. Just a quasi-creepy half smile and my best attempt at smiling with my eyes (see smizing).

Of course none of what I just said was true. As the Chief Design Officer at PolicyGenius, I did, in fact, design every screen they are about to experience, and I am in no way independent.

So why the lies? Well it turns out that the good-natured folks who are generous enough to offer up their time and insights are also too nice to tell me, to my face, that my designs are awful. God love 'em, they just don't want to crush this designer's hopes & dreams by looking perplexed at a screen I may have poured hours into. Instead, they'd rather smile back at me and deflect the awkward moment with, "Of course! That makes sense now" or, "I see - that's a button!" That type of constructive feedback, which usually rears its head in the from of general confusion, inaction, or excessive questions, is better delivered when the well-intentioned user assumes I have no skin in the game.

I call this "host protocol" - or the act of acquiring a completely neutral identity while conducting a user test. The objective is to make the users feel as comfortable as possible, and to give them free reign to bash, critique, question, or generally malign our designs. PolicyGenius is certainly not the first team to adopt this tactic. Michael Margolis, the UX research partner at Google Ventures, describes the interview process as being centered on making the user feel at home and at no risk of "hurting your feelings."

From Paul Boag's blog -

And that's just it - you'd think what's at risk is having your designs, the interface for the entire site, completely torn apart before your eyes by a pack of insurance-hungry users with an endless-thirst for the perfect insurance-shopping UX.

And you'd be wrong.

I can tell you on good authority that what transpires during these interviews, the true purpose behind all of our testing, are the little insights, the tiny fireworks that go off in your brain when you see a user stumble on something you assumed was straightforward. The end result is inevitably a string of revelations, on all scales, that make the pages more intuitive and cleanly jerk your head out of the ostrich hole. I've never left the room with a frayed nerve, or a bruised ego.

In my role as the head of design and UX at PolicyGenius, I can definitively say these interviews are the most important part of my week. It's worth the House of Cards- style maneuvering to see our tweaking and iterating pay off. In fact, take our site for a spin and let me know what you think! Email me your feedback at