Things seem blissfully simple in the beginning. You just get each other. You share a vision for the future and it’s going to be everything you hoped it would be. This is the beginning of a beautiful…survey.
Writing a survey is much like trying to meet your soulmate, amiright?
Stick with me for a moment. Dating and survey writing are much more similar than you probably think. While there are many ways a relationship or research project can go horribly wrong, there are three common lessons that are well recognised in dating, but are often overlooked in the equally ‘pupil-dilating-heart-racing’ thrill of survey writing.
Don’t get catfished. Are you sure you know who you’re talking to?
Outdated pictures. Filters. Photoshop. When you aren’t meeting face to face, it can be hard to be sure if the person you’re speaking to is indeed who they say they are. If you’ve ever seen the movie Catfish, or the TV spin-off of the same name, you’ll know that there are many reasons to be afraid.
Knowing who you’re talking to is equally also important in quantitative research, and so often overlooked. Won’t somebody please think of the screenouts! Far too many research reports conclude with sweeping generalisations about the entire market for a product or service while neglecting to be clear about who exactly the results are referent to. Even worse, this research design issue can be so subtle that even the researcher themselves may not be aware of the profound implications for the project conclusions.
Once the final questionnaire is complete, a good habit to get into is to write down the exact description of the respondent you are speaking to. For example, “We interviewed non-industry, main or joint grocery buyers living in Australia, aged 18-64, who have purchased and consumed cheese from a supermarket in the last 3 months, and purchased cheese from a specialty cheese store / deli in the last 12 months.” It is often surprising how long this list can get once you stop to write it down. The consultant and client can often recall the top-line that we spoke to those purchasing cheese for self-consumption, but qualifiers such as the latter “specialty cheese store” requirement tend to be forgotten once a report is written, and their natural incidence is very rarely provided in a debrief. If we were testing a new supermarket retail cheese offer in this survey, this one screener may have narrowed our results to a very small sub-set of the cheese buying population. While this may in fact be the intended population of interest, it’s extremely important to keep the screen-out data here, to be able to report back the natural incidence (i.e., “Of the 30% of cheese buyers / consumers we spoke to……..X, Y, Z”) so that the business can make an informed decision about return on investment.
Don’t give too much away, too soon. Don’t TMI your way out of a good thing.
We’ve all been there. The awkward silence of one person having moved a bit too quickly in the relationship, and the other not knowing how to respond. There’s a natural order to how we progress in relationships of all types; the kind of information that we choose to share upfront, and then the information that we gradually leak into conversation as we become more trusting of the other person, such as the 500 item-strong collection of Mariah Carey paraphernalia that you insist will be worth something one day.
Surveys are similar. The way that you choose to provide respondents with information has a direct impact on the answer that you elicit. I previously authored an article for AMSRS on the placement of key metrics in surveys; to summarise, best-practice would have them placed at the beginning of the survey to ensure they generalise to the broader population by not biasing the respondents ‘natural state’ based on the other questions they complete prior. Applying this logic to questionnaire writing more broadly, researchers need to carefully determine the correct ordering of survey questions to ensure that subsequent questions are not biased by information or prompts given by prior questions (i.e., ‘priming’). For example, asking about a respondent’s favourite cat memes immediately prior to asking them to rate how funny they believe the internet to be in general.
It’s not always about giving away too much though and sometimes you can give away too little. There’s no point testing take-up of a packaging innovation by placing it on a fake ‘shelf’ in a survey if the client is planning on spending millions on an educational campaign about the benefits of the new format. Here you might keep a sub-cell of respondents to test take-up post exposure to the anticipated campaign. This way the client can understand the varied response of non-exposure, and exposure to the campaign.
Put in the hard yards in. A solid foundation can help you with tricky situations down the track.
What ever happened to courtship? The world of dating these days can be fairly brutal, often involving selecting potential partners from a list of over-filtered selfies of people in athletic wear, standing in front of mountains, holding a puppy and inevitably wearing a pair of sunglasses (everyone looks good in sunglasses). We use overly simplistic selection criteria to choose our potential dates, rather than spending the time to really get to know the person. Superficiality aside, when asking couples in long-lasting relationships about the key to their success, they all have in common a mutual understanding built upon friendship that can withstand all of the unexpected situations in life.
As in dating, quantitative researchers should work to build a solid foundational relationship with their analytical techniques. While a research project might not end quite so dramatically, the impact on a client’s business can be disastrous if unguided analytics steers decision-makers to an incorrect, or sub-optimal solution. Market research as a field in particular has few barriers to entry, with many researchers from disparate disciplines acquiring a working knowledge of quantitative method on the job. While a hands-on knowledge of how to run a ‘driver analysis’ (for example, which buttons to press to get a result) generally goes unquestioned, I’d challenge quantitative researchers to at least understand the conceptual fundamentals of what’s going on behind the user interface of analytic software packages. It’s this understanding of the technique that will help you write better surveys in foreseeing the way that the questions will come together to be used in multivariate analytics, and assist you with selecting and justifying your choice of technique as a consultant, as well as being able to confidently present the results.
These lessons can help you avoid commonly overlooked errors of survey writing, or perhaps even to find love*.
*The author regrettably has no formal qualifications, expertise or even sufficient life experience in dating and all suggestions relating to the quest for love should be taken with a grain of salt, or ignored entirely.
Author: Amy Tildesley, Founder of Harvest Insights, a food & beverage market research consultancy helping brands design the products and services that their consumers want and need.