How to ask your questions
You’ve thought carefully and critically about what you need to ask and now it’s time to write your questions. Writing good survey questions is a skill and it takes time to learn the things to be cautious of, but once you know them ... you will critique every survey you take!
-
Watch for leading questions. Keep the language of questions as neutral as possible. Leading questions are clearly trying to impact how participants will answer the question or restrict how they’ll answer. Leading is naughty.
-
Check your assumptions. You can’t assume your participants do or know things; this impacts their ability to answer truthfully.
-
Beware of the double-barrel. Two questions in one is asking for trouble! A question that uses AND or OR is the telltale sign.
-
Eliminate bias (positive and negative). While this is similar to leading, bias is more about skew or prejudice in how the question is written than trying to push participants in a specific direction.
-
Avoid using absolutes (always, never, etc.). This gets back to whether the participant can truthfully respond to your question. Do you always brush your teeth before bed? Absolutes are ok in response scales, but should be avoided in your questions.
-
Know your audience and use clear language. Watch out for acronyms, technical terms, or jargon that may confuse your participants.
-
Be cautious of binary response options and limiting your participants choices. You don’t want to force your participants to put themselves a box. A well known example for this is only offering Male and Female response options for a question about gender.
Another element to being thoughtful about how you ask your questions is how you structure them within the survey itself. Thinking more broadly about the design of your survey means considering things like what order you put the questions in and how many pages your survey is. Some tips:
-
Put like questions together. Chunking is a really useful for breaking content down concept-by-concept. With surveys, consider grouping questions around the same concept together so you’re not asking participants to jump between concepts.
-
Be cognizant of how the order of your questions could impact the way people answer. Think carefully about question sequence. This can get down to logic (a question may make sense leading into others), as well as considering whether a question could trigger emotions, memories, etc. that might affect how questions that follow are answered.
-
Consider how to help your participants using layering and pagination. Is your survey simple and short enough to all be on one page or do you need some element of progressive disclosure? Can you help people focus on the chunks of questions by having each on their own page?
-
If you do paginate, I highly recommend turning on whatever option your tool has for displaying progress. This is important for keeping expectations clear to your participant so they know how much of the survey is left.
-
-
Explore conditionals when it makes sense. I kind of love conditionals even though they can add a lot of complexity. If you need to know different things based on how someone responds to a specific question or questions, conditionals are very powerful and prevent you from having to present questions to participants that are not relevant to them.
What types of questions to use
When you do so much work to thoughtfully design your survey, it would be a shame to fall off the wagon and pick totally wrong question types. So ... think just a little bit more!
The main distinction is between open (text fields) and closed (multiple choice and all its friends) questions. And there are two equally important things to consider:
-
How long it will take participants to complete the survey. The more you ask, the longer it takes. And open questions take longer to answer.
-
Your capacity for analysis. Open questions take much more time and effort to analyze. They are also more susceptible to your biases.
Open questions
It can be really tempting to throw a lot of open questions into surveys. I think it’s human nature to want that sweet, sweet context, but I encourage you to think about whether you can ask an open question in a way that fits with everything already mentioned here. Is it a question a survey can answer? Is the question specific enough to analyze effectively? Are you prepared to do follow-up research with more intensive methods? Keep in mind that asking open questions often reveals new things you’ll need to explore.
The types of open questions are pretty straightforward: little text field or big text field. I never used to give this much consideration, but it’s helpful to think about what message you’re sending with the one you choose. How much do you want to invite people to write? How much do you expect them to write? It stinks when you’re given a tiny box and need more room to adequately answer the question. And it can be confusing when you’re given a big box for a basic question, since it can feel like you’re missing something by not having more to write.
Closed questions
There are copious options for closed questions. You need to think through what is going to work best for the question you are asking. You should also think about ease of use of those question types. There are some survey questions types out there that are a real challenge to use. Don’t make your participants think about how to respond to the question! You want them focused on their answer.
Some of your most prominent options include:
-
All your flavors of multiple choice. The biggest variable there is thinking through whether you allow participants to choose only one response or more than one.
-
More complex options like rank order, linear scale, side-by-side or matrix questions, pick, group and rank, and things like pre-formatted fields (e.g. date, time).
My advice, which is straight opinion, is to keep the thing as simple as you can. Try not to use too many different types of question formats. It all gets back to cognitive load and how much our brains can handle. And for goodness sake, have someone (or many someones) test your survey before distributing it.
Practice and use your new powers for good
That’s my story. Like I said ... lot of feelings coming from me about surveys. If you’d like to practice your new skills, check out this Mainstream Media Accountability Survey. Then read this great post from NPR, and let it remind you about how powerful surveys can be. I hope you’ll use your survey creating powers for assessment, evaluation, learning, and overall good.