Data Tips for Marketers Part 2: Getting Better Survey Data

By: Tom Swanson, Engagement Manager at Heinz Marketing

In my last post, I talked about some methods of automatically cleaning survey data to fix a few common usability issues that come from most survey tools.  While useful, the functions listed there are for triage on data sets that have issues, but what about how you discover issues in the first place?  How about avoiding them entirely?  In this post, we will go over some tips on what to look for that could cause issues in your survey data sets.

Tip 1: Have a sampling & analysis plan

Ok, I get it, this sounds obvious, but it is surprising how many folks get caught up in things like wording, completion time, goals, and promotion.  Analysis plans often get left in the dust, but they are very necessary at the design-stage of the survey, not after you have your responses.  A good sampling & analysis plan includes, at least, the following:

• Project goals
• Project timeline
• Project team
• Sampling plan
  • • Data gathering objectives (Response #’s, target respondent, etc…)
    • Data review and validation procedures
    • Data storage procedures
• Analysis plan
  • Objectives
  • Filtering strategy (see note below)
  • Methods (how will the data be analyzed to hit the goals?)
  • Step-by-step plan along timeline

A note about filtering: much like market segmentation, your filtering strategy is how you will segment your survey data to get at meaningful insights.  At the same time, you need to keep question-count as low as possible so people will actually complete it.  You need to know exactly what sets, subsets, sub-subsets, etc… that you want to see before the survey hits the air and get at those filters as efficiently as possible.

Tip 2: Quantify as much as you can

Context matters, and nothing gets at context like qualitative data, but that data comes at a significant cost.  It is subjective and messy, time-consuming for everyone involved, and generally just harder to analyze (for a few reasons like spelling/grammatical errors).  As I mentioned before, qualitative data can be great for the odd quote and contextual question, but it is often a better use of time to quantify as much as possible.  There are two times to do this:

Before the survey

This is, by far, the easier way to do it.  Being judicious about where you need contextual information is key.  Whenever something could be a list of pre-determined options, it likely should be.  You may ask respondents to describe their experience with your brand, but a better option would be to ask them to rate their experience, and then present a list of adjectives for them to select from that would accomplish the same thing.  The first option would leave you with sentences that you need to purge of errors and then interpret, while the other presents a list of clean, standardized data.  It is faster for everyone and less prone to human error.

After the survey

You have a big ol’ bucket of qualitative data and are wondering how to seek trends within it. There are a few ways to do this.  The first is called “coding” the data, which is a long and rigorous process of interpreting the meaning of each response, one at a time, according to a schema and with a team of people who must establish interpretive reliability on a sample data set prior to starting.  Yikes.

The next is to use functions and other tools to scrape the data set for answers that meet certain criteria (like the FIND or SEARCH functions in Excel).  The specific functions depend on what you are trying to do with the data.  Maybe you want to search only for responses that use the word “ghost” or “vampire”.  Then you will need to pull those responses and conduct the analysis required for your insights.

To put it shortly: qualitative data is impractical to analyze at scale, and is not often going worthwhile for marketers.  You need to be the judge of this for yourself, but here is when I include open-ended questions:

• I want quotes to include in a report or content piece
• I am looking to source creative ideas
• I need a hurdle to weed out low-effort survey responses
• My survey is 1:few and won’t need large-scale analysis

I am sure there are others, but this is where I usually wind up seeking this form of data.

Tip 3: Use paid panels

Here is a great idea: if you want to drive large amounts of involved survey responses, just include an exciting incentive!  Genius, right?  Not so fast.  Bots and spammers figured this one out a long time ago, and an open, incentivized survey is an excellent way to attract them.  There are numerous telltale signs of bot/spammer infiltration (we will get to those next), but one thing is for certain: if you offer incentive, they will come.

On the flip side, surveys with no incentive barely gather any attention at all.  Unless you are at one of those rare companies with energized fans that are excited to give their opinions, you are unlikely to get much traction.  Kudos to those community managers who can build that, but for many B2B companies, and for many surveys, this just isn’t a great option.

So, how do you get useful survey responses?  Take the incentive budget and put it towards getting a paid panel.  Third party firms maintain a healthy database of qualified people defined by demographic, psychographic, professional, and many other attributes.  Almost every survey platform is going to offer this service, and even if your platform doesn’t, you can find companies that will also distribute your survey.  You will get guaranteed volume and a high level of quality

Tip 4: Read the data set

 

Personally, I love digging through raw data, but it is not for everyone.  Most survey tools offer some sort of analytics dashboard that will slice and dice your data for you and show all the exciting insights.  It is intoxicating to see the fruits of your survey laid out so clearly and easily before you.  Don’t fall into this trap.  Even if you go numb when staring at unending rows and columns, you must still wade into the actual responses, in their spreadsheet form, and look through it.  These dashboards are useful but relying on them exclusively can hide data cleanliness issues from you and limit insight you can gain to what is shown in the interface.

Getting curious and digging through the raw data will do few things:

• Help you catch and remove non-viable respondents (bots/spammers)
  • Look for response patterns (e.g. every answer is C) or clearly auto-generated email addresses (random numbers/letters)
• Show you where you need to clean the data
  • See my previous post on “select all that apply” questions for an example
• Give you far greater flexibility as you analyze the results
• Show you how you could structure future surveys for better analysis
• As an added bonus, it will make you better at planning and cleaning future data efforts

Click that “download as .csv” button and get started.

These are just a few tips to get you going on designing and implementing more organized and effective surveys.  There are plenty of other resources that can enhance your ability to create useful and easily analyzed survey data.  Additionally, think about what data source is right for what you are trying to do.  Surveys are great for when you have a specific question or goal for your audience, but not always so great when you are looking for more general trends and ideas.  Speaking broadly, surveys are best for testing discrete hypotheses, and as with the qualitative data point, the more specific you can get with them, the better.

As always, if you have any questions, feel free to reach out to me at tom@heinzmarketing.com.