[Off-Topic] Most people said they'd use my product. Will it work? No!

First published on 2013-03-13T14:06:39+00:00

“We’re putting together an excellent new product that will solve your problems, would you use it?” says the validation survey of a new startup. Result: 70% of the 1,000 people who answered said yes. So the conclusion is clear: the product is valid and can’t go wrong, right?

Wrong. Those numbers don’t mean a thing. On their own they carry no validity, and they support no conclusion at all. “But it’s 1,000 people! How can you say that?” This article isn’t trying to be statistically rigorous, just to point out a few of the many places where almost everyone gets it wrong when running a survey these days.

First of all, what’s your product’s target audience? Let’s say it’s the population of the interior of the southern states, classes C and D, ages 50 to 60, with little experience using computers. Now imagine the question above was part of a survey done online, shared on Facebook and Twitter.

Who answered it? Plenty of people from classes A through C for sure, concentrated in the southeastern capitals, ages 16 to 25, on a computer every day. You asked the wrong question to the wrong people.

For the sake of the example, let’s say you took care to pick the right population and dodged what’s known as “Biased Sampling.” Even so, the question is still bad, and it should be obvious why. It’s what we call a “Loaded Question”: the wording already nudges the respondent toward a certain answer.

For starters, it takes for granted that the product is “excellent” and that it “will solve” the person’s problems. A less loaded form would be: “We’re building a product with features X; assuming you have problems Y, do you think this would help solve them?”

There are plenty of techniques for steering clear of baseless premises, like actually explaining the value proposition instead of stuffing the question with rhetoric. It’s very easy to build a fallacious question that pulls a false answer out of people.

It gets worse when you take those 1,000 people, single out the 700 who said “yes” to the rigged question asked of the wrong audience, and start drawing conclusions from it. “Oh, we noticed that of these 700, half have a credit card, shop online at least once a month, and say they’d share the service with at least 2 friends on social media.” And it gets worse still when the ones who said “no” happen to be the audience closest to the target you want to reach.

I also consulted Ricardo Couto* who explained the issue. “Knowing the respondent, their habits and motivations is much more valuable for the business than having a high percentage of favorable responses about the product or service in an online questionnaire. By the way, calling a form put together in 5 minutes on Google Docs or SurveyMonkey a ‘survey’ is, at best, naïve. What you have there is a questionnaire. Questionnaires can be useful if used well. But in general, they’re just sets of questions around a subject. A survey can even make use of that tool, but not every questionnaire is a survey.”

Statistics, probability, and research in general are the work of professionals who master this field. It’s no trivial matter, and it’s not something to handle carelessly. This example is just one of the many ways to fool yourself with numbers.

Mark Twain popularized the phrase “There are three kinds of lies: lies, damned lies, and statistics.” The subject has been discussed and rebutted so widely that it produced a famous 1954 book by Darrell Huff, “How to Lie with Statistics,” which catalogs the intentional and unintentional errors in interpreting statistics.

Ricardo also added: “If we think of an online questionnaire as a serious research tool, first you have to know how research is done. I really doubt that the people who say they ‘are doing’ a survey and send you a link to a questionnaire took into account the order effect, the observer effect, or avoided leading questions – factors well known to anyone who does real research. And I’m not even getting into statistical validity, which you only reach by applying statistical methods (a simple percentage isn’t descriptive statistics, okay? you have to know what a p-value is), or into selecting respondents suited to the target audience you intend to reach. Or do I only want, as users, customers, or consumers, professionals in my age bracket, my income range, and my online shopping and social media behavior?”

So the next time you go to create an account on one of those dozens of online survey services, check whether the tool meets the bare minimum of a well-done survey and walks you through the right steps: defining the audience, collecting, processing, and analyzing the data rigorously. Without that, all you’re left holding is a pile of opaque numbers, with no meaning and no conclusion to draw from them.

*Ricardo Couto is a Consultant in User Experience and Research, Mentor at Aceleratech, President of the Interaction Designers Association SP (IxDA SP), and Member of the Brazilian Information Design Society. He studied Cognitive Psychology (master's) at the Federal University of Pernambuco and specializations in Information Design, Higher Education Teaching, and Distance Education Technology. He was a university professor at undergraduate and graduate level for 7 years.