Designing a Survey: Questions, Scales and Piloting

Surveys look easy: write some questions, send a link, collect answers. Then the analysis begins, and you find that half the questions cannot be interpreted, that people skipped the important ones and that your scale makes every answer cluster in the middle. A well-designed survey prevents these problems before they start.

This guide walks through survey design step by step, from deciding what you need to measure to piloting and distribution. It uses invented examples to show good and bad wording.

What a survey can and cannot do

Surveys are good at describing attitudes, behaviors and characteristics across many people, comparing groups and looking at relationships between variables. They are weaker at explaining why, capturing complex experiences and showing cause and effect. Their quality depends on three things: the questions, the sample and the response rate. Our guide to sampling methods explained covers who you ask, and this guide covers what you ask.

Step 1: Start from your research questions

List your research questions and hypotheses. For each, decide what you need to measure and how. Draw up a table.

Every question in the survey should map to a row in this table. If a question does not serve a research question, remove it. Surveys balloon when they collect interesting but unused data.

Step 2: Check for existing instruments

Many concepts, such as stress, belonging, burnout or satisfaction, have established questionnaires that have been tested for validity and reliability. Using one saves time and strengthens your study, since you can compare your results with earlier work. Search the literature for scales used in similar research, check the terms of use and cite the source. If you adapt a scale, say how.

Step 3: Choose question types

Closed questions are easier to analyze, and open questions add richness. Use a few open questions where you need explanation, and keep them optional so people do not abandon the survey.

Step 4: Write clear questions

Wording shapes answers. Follow these principles.

• Use plain language. Avoid jargon and abbreviations.
• Be specific. "How often do you study?" is vague. "In a typical week, how many hours do you spend studying outside class?" is clearer.
• Ask one thing at a time. Avoid double-barreled questions.
• Avoid leading or loaded wording. "How much do you enjoy our excellent advising service?" pushes a positive answer.
• Avoid negatives and double negatives.
• Give a time frame where relevant, such as the past week.
• Make response options complete and exclusive. Include "other" or "not applicable" if needed.

Step 5: Choose response scales

For rating scales:

• Use a consistent number of points, commonly five or seven.
• Label every point or at least both ends clearly.
• Choose a midpoint deliberately. Odd scales have a neutral option, even scales force a choice.
• Keep direction consistent, so that higher always means more, or more agreement.
• Match the scale to the question, such as frequency (never to always) or agreement (strongly disagree to strongly agree).

Avoid mixing many different scales in one survey, which tires respondents and complicates analysis.

Step 6: Order and structure

• Start with an introduction that explains the purpose, time required, consent and contact details.
• Begin with easy, engaging questions.
• Group related questions under short headings.
• Place sensitive questions later, once rapport is built, and make them optional where appropriate.
• Put demographic questions near the end, unless they are used to screen participants.
• Keep the layout clean, with consistent formatting.
• End with thanks and information about what happens next.

Be aware that earlier questions can influence later ones, so think about the order.

Step 7: Keep it short

Every extra question reduces completion. Ask whether each item is essential. Cut anything you are not going to analyze. Test the time it takes, and tell respondents the honest estimate. For student projects, 10 to 15 minutes is a usual upper limit, and shorter is usually better.

Step 8: Pilot the survey

Before you launch, test the survey with a small group similar to your respondents, such as five to ten people. Ask them to complete it, and then ask:

• Were any questions unclear or confusing?
• Did any options not fit your answer?
• How long did it take?
• Did anything feel intrusive?
• Did the survey work on a phone?

Revise based on the feedback, and test again if you made big changes. A pilot is the cheapest way to catch problems. Report it in your methodology.

Step 9: Choose a mode and a tool

Surveys can be online, on paper, by phone or in person. Online surveys are easy and cheap, and tools such as Google Forms or other survey platforms offer templates. Check that your university allows the tool and meets data protection requirements. Paper surveys may reach people who are not online. Whatever the mode, make the survey accessible: readable fonts, clear contrast, screen reader compatibility and mobile-friendly layouts.

Step 10: Raise the response rate

Low response is a common problem. These steps help.

• Explain why the survey matters, and who is behind it.
• Keep it short and easy to complete.
• Send a clear invitation with a personal tone.
• Send one or two reminders.
• Ask a trusted gatekeeper, such as a lecturer or manager, to endorse it, if appropriate.
• Offer an honest incentive, where your ethics approval allows it, such as a prize draw.
• Make it easy to open and complete on a phone.

Record how many people you invited and how many responded, so you can report the response rate.

Validity and reliability

A good survey measures what it claims to measure and does so consistently.

• Validity: do the questions capture the concept? Use established scales, expert review and pilot feedback.
• Reliability: do multi-item scales hang together? Researchers often report a statistic such as Cronbach's alpha for scales, which shows internal consistency.
• Common method issues: self-report can be affected by memory and social desirability, so word sensitive questions carefully and assure confidentiality.

Ethics and privacy

Surveys involve people, so ethics approval is usually needed. Plan for:

• Informed consent, shown at the start.
• Voluntary participation, with the freedom to skip questions or withdraw.
• Anonymity or confidentiality, with a clear explanation.
• Data security, including where data are stored and who can see them.
• Sensitive topics, with information about support services.
• Minimizing data, collecting only what you need.

Our guide to research ethics and ethics approval explains the process.

Preparing for analysis

Plan your analysis while you design. Decide how each question will be coded, what tests you will use and how you will handle missing data. A short mock analysis with pilot data can reveal problems, such as a scale that every respondent answers the same way. Our guide to data analysis basics for students introduces the basics.

Common mistakes

• Writing questions before defining what to measure.
• Double-barreled, leading or vague wording.
• Too long a survey.
• No pilot.
• Inconsistent scales.
• Non-exclusive response options.
• Collecting data you will never use.
• Ignoring the response rate.
• Weak privacy and consent information.

A quick checklist

• Does every question map to a research question?
• Have I checked for existing validated scales?
• Is the wording clear, neutral and specific?
• Are scales consistent and fully labeled?
• Is the survey short, logical and accessible?
• Have I piloted and revised it?
• Do I have ethics approval and a consent statement?
• Do I have a plan for analysis?

Final thoughts

A good survey is built backward from the analysis: know what you need to measure, ask each question as simply and fairly as you can, test it on real people and keep it short. The work you put into design is the work you will not have to do later, trying to rescue unusable data.