Chapter 2: Developing a Research Study
Module 4: Designing Survey Questions
Designing survey questions can seem like a daunting task for researchers, especially for those who are doing it for the first time. This module will explain how to write a survey questionnaire, common survey platforms and terminology considerations.
Learning Objectives
- Learn the key steps in designing inclusive, thoughtful survey questions
Case Study
Ethan is creating a survey to understand barriers to non-OHIP-covered healthcare access. He includes a question asking, “Do you have a disability?” with a simple “yes/no” response, as his demographics disability question.
After launching the survey, Ethan received feedback that some participants found the question was not inclusive and hard to define. In a conversation with a colleague, Ethan shared his confusion, saying, “Isn’t that question as straightforward as it should be? What other options could I add?”
What questions do you need?
The first thing to consider when developing questions is asking yourself what types of questions you need.
The demographics section of your survey is a great place to start. Who is your target population? What demographics (age, gender, sexuality, marital status, employment, race, ethnicity, etc.) will be important to inform your data? What types of correlations do you want to draw between your dependent variable (what you’re studying) and these demographics?
Next, what is the main question guiding your research? What do you want to find out? Once you answer that… What types of survey questions do you need to ask your participants in order to answer that main question?
If you aren’t sure where to start, this is where a literature review, consultation with your peers, and co-design come in handy.
Preliminary Research for Question-Writing
All science and knowledge seeking is iterative. That means that discovery is a continuous process of development and improvement, which is central to research in every area of study. This cycle allows for the refinement of existing ideas and development of new ones (Utah Valley University, 2024).
In survey research, we know that there are millions of existing survey questions that have been answered by millions of people- many of which may be relevant to your own research questions. Saturate yourself with existing literature about your topic, and especially finding other surveys that have similar premises to yours, will give you a guideline for types of questions to ask, and how to ask them.
For example, for collecting demographic data from persons with disabilities in Canada, you might consider looking at previous Statistics Canada surveys.
The Canadian Survey on Disability, 2022, used some of the following questions to document gender:
While this is a poor example of an inclusive survey question, you can still use it as a reference. You may choose to use similar questions to ask about gender. You may also decide that you would prefer to modify the question, in order to incorporate additional options aside from the binary male and female genders (we would certainly recommend this).
Another example from the same survey is regarding disability assessment. These questions are clear, concise, and can be understood in the same way by all participants; they might be a great place to start if you wish to ask similar questions:
Consultation with experts and doing a co-design session with your population of interest is also vital to ensuring that your questions encompass the scope of values and knowledge that you wish to gather.
The Fundamentals of Survey Question Writing
So now you know what kinds of questions you want to ask… How do you ask them in an effective way?
Let’s start with the basics! Writing survey questions is all clarity. That means that your questions need to be simple and direct. A poorly worded question can lead to confusion, misinterpretation, or even the exclusion of certain participants. This is why crafting clear, inclusive, and accessible questions is essential, especially when working with people with disabilities, racial and ethnic minorities, and other groups that commonly are subject to structural/systemic marginalization.
Generally, a survey question should satisfy the following criteria:
- It captures the specific idea or behaviour being studied.
- It excludes all unrelated topics.
- All participants understand the question the same way.
But how do we achieve all of these things? There steps that can be followed in order to keep to these standards. To simplify this, you can follow the mnemonic:
Concise: Keep questions short and simple.
Language: Use plain, inclusive, and accessible language.
Exclusive vs. Inclusive: Avoid double-barreled questions and offer diverse response options.
Avoid Bias: Don’t lead participants to a particular answer.
Review: Always pilot your survey before finalizing.
Concise (keep it short)
Ensure your questions are simple and short. Long-winded writing and overly complex questions can be intimidating and demotivating for participants.
Language (simple but specific)
You are the expert, but your participant is very likely not! Always try to use the most “plain English” version of what you are trying to convey– avoid overly technical or professional jargon that the general public are unfamiliar with.
In the same way, avoid overly vague language. Vague language can frustrate readers and leave too much space for misinterpretation. Be specific with your questions. For example, “Are you happy with life?” is vague, and many readers will understand it differently. Using a concrete measure can make this question more specific, such as a time frame or a location.
Furthermore, when asking measuring questions, make sure your chosen scales are ordinal and precise. “Many, a lot or a few” would be inappropriate because what is “many”? What is “a few”? Is this scale in ascending order? A better example might be “none at all, a little or many.” If possible, including figures and units of measurement is helpful.
Exclusive vs. Inclusive
You need to ensure that your question is clear, and only asking one thing. Do not include “double-barreled” questions, as they may also confuse readers. These questions often include “and,” & “or”. Thus, a question like “Is the sky blue and pretty?” should be split into two questions: “Is the sky blue?” and “Is the sky pretty?”.
While you want your question to be clear and simple, if you are providing pre-selected possible responses, such as in a multiple choice question, these possible responses need to be diverse and relevant to your target population. Offering diverse possible responses to questions is important to ensure that the data is not being biased towards the “least inaccurate” response.
For example, many demographic questions are exclusive of important responses, such as gender-based questions. Including only the binary male and female as possible genders, results in an exclusive question against those who do not conform to the binary genders.
Avoid Bias (don’t lead!)
A biased question, often labelled as a “,” refers to a question that guides a participant toward a specific answer. This can be done using specific adjectives or phrases. Overly emotional or evocative language can also lead to biased answers.
For instance, “Do you believe that the controversial sales tax is good?” would be a leading question because the reader may change their answer due to the word “controversial”. Avoid introducing any bias into questions to ensure the most representative answers from participants.
Review (throughout)
This step is crucial: review your work! In the same way you might proofread your paper before submission, you must review your survey questions. Reviewing your survey questions alone carries the risk of bias, so using outside sources, such as co-researchers or willing community members, to help is an ideal way to review your work.
While the final step for survey question development is “Review”, we recommend that you undertake review as a part of the entire design framework of your research, rather than just a final step of survey question development. Co-design is important throughout question development, even for quantitative data collection and for demographic information.
As a researcher, you have the power to incorporate the voices of the community in the entire process of developing research. Providing opportunities to review and pilot the survey after the main design process is a good start, but consider co-design as a better option. This ensures that we create research “with”, rather than solely “for” populations.
Module 4 Question 1
Concise: Keep questions short and simple.
Language: Use plain, inclusive, and accessible language.
Exclusive vs. Inclusive: Avoid double-barreled questions and offer diverse response options.
Avoid Bias: Don’t lead participants to a particular answer.
Review: Always pilot your survey before finalizing.
A question which unduly guides a participant toward a specific answer. For example, "How do you feel about our excellent education system?".
Feedback/Errata