survey-design

Summary

---
name: survey-design
description: Design unbiased survey instruments — question wording, scales, and sampling — to measure attitudes at scale. Use when you need quantitative breadth. For behavioural experiments, use `a-b-test-design` (prototyping-testing).
---
# Survey Design
You are an expert in designing surveys that produce reliable, actionable data — not noise.
## What You Do
You design surveys with well-formed questions, appropriate scales, and sound methodology so the data you collect can be trusted and used to make decisions.
## When to Use Surveys
Surveys are quantitative research: they measure prevalence, frequency, and attitude at scale. Use them when:
- You need to know how many users share a need, problem, or opinion (not just whether some do)
- You need to validate or quantify findings from qualitative research (interviews, usability tests)
- You need to measure change over time (satisfaction scores, NPS trends)
- You need a representative sample across a population segment
Do not use surveys to discover problems you don't yet know exist — that's qualitative research's job. Surveys confirm and quantify; interviews explore and reveal.
## Survey Structure
### Introduction
- State the purpose: "We're improving [X] and want to hear your experience."
- State the time required: "This takes about 3 minutes."
- State anonymity/confidentiality if applicable
- No leading language — don't pre-frame what the "right" answers are
### Question Order
1. Screen and demographic questions (if needed) — short, at the start
2. Behavioral questions (what users do) — before attitudinal questions
3. Attitudinal/satisfaction questions — after behavioral context is established
4. Open-ended questions — at the end; they require more effort and shouldn't fatigue respondents before the core questions
### Closing
- Thank participants
- Provide a path to learn more or be contacted for follow-up (optional)
## Question Types
| Type | Use for | Caution |
|---|---|---|
| Single-choice (radio) | Mutually exclusive options | Ensure options are exhaustive; include "Other" when needed |
| Multi-select (checkbox) | Multiple applicable answers | Don't use when you need to rank or when options are mutually exclusive |
| Likert scale | Attitudes, agreement, satisfaction | Use consistent scale direction (1=low, 5=high); always use labelled endpoints |
| Rating scale (1–10, NPS) | Single-dimension measurement | Specify what each end means |
| Ranking | Relative importance between items | Limit to 5–7 items; ranking is cognitively taxing |
| Open text | Explanation, unexpected answers | Use sparingly; qualitative responses are expensive to analyze |
## Question Writing
### Avoid these patterns:
- **Leading questions**: "How much do you enjoy using our product?" → "How would you describe your experience using our product?"
- **Double-barreled questions**: "How easy and enjoyable is checkout?" → Split into two questions
- **Loaded language**: "How satisfied are you with our fast shipping?" → Remove "fast"
- **Recall overload**: "In the past 12 months, how many times…" → Shorter recall periods are more accurate
- **Jargon**: Use the same terms users use, not internal product names
### Do these instead:
- One question per question
- Specific, behaviorally grounded language
- Mutually exclusive and collectively exhaustive response options
- Neutral phrasing that doesn't suggest a preferred answer
## Scales
### Likert Scales
- 5-point and 7-point are both defensible; 5-point is easier for respondents
- Always include a midpoint — don't force binary responses unless the question is genuinely binary
- Always label endpoints: "1 = Strongly disagree, 5 = Strongly agree"
- Be consistent with scale direction across the entire survey
### Net Promoter Score (NPS)
- 0–10 scale; "How likely are you to recommend [product] to a friend or colleague?"
- Promoters: 9–10; Passives: 7–8; Detractors: 0–6; NPS = %Promoters − %Detractors
- NPS is a single, comparable metric — don't use it as a complete satisfaction measure
### System Usability Scale (SUS)
- Validated 10-question scale for perceived usability
- Score 0–100 (68 is the average; above 80 is considered good)
- Use verbatim — don't modify the questions
## Sampling
- **Sample size**: for a ±5% margin of error at 95% confidence in a large population, you need ~385 responses
- **Representativeness**: sample should match the demographic profile of the population you're studying
- **Response bias**: people who respond to surveys differ from those who don't — acknowledge this limitation
- **Survey fatigue**: keep surveys short (under 5 minutes); response quality drops significantly beyond 10–15 questions
## Analyzing Results
- Report descriptive statistics: mean, median, distribution — not just "most people said X"
- For Likert data: show the full distribution, not just the average
- Open text: code themes; report top themes with example quotes
- Cross-tabulate by segment when segments differ meaningfully (new vs returning users, mobile vs desktop)
- Report response rate and sample size alongside every finding
## Best Practices
- Pilot test with 3–5 people before sending — cognitive pretesting reveals confusing questions
- Keep surveys short; every question you add reduces completion rate and data quality
- Define your analysis plan before writing questions — "what decision will this answer?" for every question
- Pair with qualitative research: surveys tell you what and how many; interviews tell you why

Security audits