What to do when you can’t trust your customer feedback
Yesterday I read a great article called What You Should Never Ask Your Customers by David Bradly.
The article talks about how asking the wrong questions during customer interviews and user research leads to bad answers and a product catastrophe. It also mentions the moms test, a term I wasn’t familiar with, which is based on the book The Moms Test by Rob Fitzpatrick.
The moms test:
“Everyone is innocently lying to you all the time, for a wide variety of reasons and the person who lies to you most is your mother, because she loves you and doesn’t want to hurt your feelings.”
(if you’re like me, you’re probably already falling in love with this brilliant premise…)
Failing the moms test is easy: all you need to do is ask your customers some really bad questions:
1. Ask generic questions such as “how often do you go to the gym?”2. Ask for someone’s opinion about your product or idea: “What do you think about…?”
3. Ask hypothetical questions such as: “would you go to the gym more often if it had…?”
I just love the moms test! Let me throw in the friends test.
The friends test:
The premise behind the friends test is that half of the things you want to do with your friends never happen because of a variety of reasons, but that doesn’t stop you from giving those empty statements and promises that never happen.
- “We should go out more often”
- “We should play tennis sometimes, it’s been a while!” (20 years…)
- “We should arrange a weekend with the families!”
- “Next time you are flying to London let me know and I’ll join you (leaving leave my 3 months baby…)”
Friends don’t lie to each other consciously. They mean what they say but their wishful thinking, combined with a total lack of self awareness, are distorting their thoughts, making it impossible to trust what they say.
We all fail the friends test, which reminds me of a story:
The (rise and) fall of our local running group
Few months ago, in one of our much unneeded neighbourhood events, my wife was persuaded by her friends to start a running group on WhatsApp. The idea was simple: everyone we know runs every once in a while. In the cold winter it’s harder to find the energy to do it, and having an extra social push from someone in the neighbourhood might help make it happen. We created a running group.
Everyone we know (well, besides a few anti-socials like myself…) asked to join this group: neighbours, parents from our kindergarten, parents from our kids’ school, etc.
In terms of early registration - the conversion rate was beyond comparison (almost everyone involved asked to join the group). We even started to believe a small app would be a good idea for this.
Reality check
3 weeks and 0 activity in the group later - it was obvious this product had serious engagement issues.
- In theory, we believed the running group had a strong product/market fit.
- In reality - we all failed the friends test, proving once again that the zombies are not coming.
Not only for new products
Getting customer feedback for an existing product can go wrong as well; some customers might give you the answers they think you are looking for while others might drag the discussion to specific feature requests or a checklist-style comparison between you and your competitors.
What to do when you can’t trust anyone
So if everybody is lying or just giving you the wrong answers, how can you get that feedback you need to build your product?
Before you have a product:
- Conduct your interviews carefully: talk less and listen more.
- Collect facts and not opinions.
- Understand what your potential customers do rather than what they say they do (or think they will do).
- Never ask: “how much would you pay for it?”, ask: “how much did you pay for... [something similar]”.
- Collect problems & pains, not feature requests (see: problem roadmaps).
When you have a product direction:
- Create a low budget MVP using existing tools like email, facebook, WhatsApp groups.
- Pro tip: get volunteers to test your potential competitors’ apps. See what works and what’s not.
- Cherry-pick your first users and early adopters. Wrong users will lead to poor decisions and poor product/market fit.
- Make your first users commit to the MVP (by paying or committing to being active) - this will help you screen out bad candidates, even if they are your friends.
- Do not count on your customers to do the visionary work for you. Problem solving is an art. Your art.
Which reminds me how 10 years ago our field service customers asked us to develop timesheets app running on Blackberry, the most popular business smartphone back at the time. Analyzing the future we rejected their request, and built our product strategy on iPhone and Android instead, resulting in hundreds of happy customers.
When your product is up and running:
- Measure every possible thing: user behavior in each session, user behavior over time, analytics based on individuals, cohorts, segments, everything you can possibly think of.
- Store every piece of information possible - believe me, your database will not collapse, but this data will be needed some day.
- Use A/B test to prove or disprove your theories without asking your customers about it.
- Your customers are nothing but numbers. Get to know them, show empathy, but take cold decisions based on their behavior only.
- Ignore vocal minority and focus on the silent majority - always prefer the big picture over local escalations.
- Make data driven decisions and analyze your decisions based on data.
Comments
https://medium.com/@asoehnlen/the-common-flaw-in-a-b-testing-17080685b353
Post a Comment