Cognitive interviewing
Cognitive interviewing tests a questionnaire before a study uses it. People answer the draft questions while an interviewer finds out how they understood each one and how they chose their answer.
It helps show whether each question gets the information you meant it to, and it is one of the main ways to find and fix problems with survey questions (Beatty and Willis, 2007).
What can go wrong when a person answers
| Step | What can go wrong | A probe for it |
|---|---|---|
| Understanding the question | A word or phrase means something else to the person | What does "in the past week" mean to you? |
| Remembering | The person cannot recall what the question asks for | How did you work out that number? |
| Judging | The person guesses, adds up loosely or answers to look good | How sure are you of that answer? |
| Answering | The person's answer does not fit the options | Was it easy or hard to choose an answer? Why? |
The four steps follow the model of Tourangeau, Rips and Rasinski (2000), who call them comprehension, retrieval, judgment and response.
Think-aloud or probing
| Method | How it runs | Trade-off |
|---|---|---|
| Think-aloud | The person says what they think while they answer each question, and the interviewer listens | Little steering from the interviewer, but some people find it hard to keep talking |
| Concurrent probing | The interviewer asks planned or spontaneous probes after each question | Focused answers, but the probes can change how the next questions are read |
| Retrospective probing | The person answers the whole questionnaire, then the interviewer goes back over it | The questionnaire runs as it will in the study, but people may not remember why they answered as they did |
Beatty and Willis (2007) review these ways of working and their apparent advantages, and note where standards for best practice are not clearly defined, so take the trade-offs as a guide, not a rule.
Rounds
- Run the interviews in small rounds. After each round, change the questions that caused trouble, and test the new wording in the next round.
- Choose people like those the study will ask, including those most likely to struggle with the questions.
- Write down, for each question, what went wrong and what you changed: the record shows why the final wording is what it is.
Who to interview, how many interviews to run and how to choose the probes are decisions each study makes, with no fixed standard (Beatty and Willis, 2007).
Web probing in ConductExperience
Web probing asks the probes in the questionnaire itself, so many more people answer them than an interviewer can see:
- After a question you are testing, add a long text question that asks how the person read it, with up to 2,000 characters for the answer.
- Rules show a probe only when an earlier answer calls for it.
- Piped text puts the person's earlier answer into the probe's words: "You chose Often. What did you count as often?"
- Results lists each written answer with the person's study code. Only people who can download the data see them.
Sources
- Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311.
- Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press.
- Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.
Related
Readability test
Flesch Reading Ease, Flesch-Kincaid grade, SMOG and the fog index of a consent form or instructions.
Face validity
Whether a questionnaire looks right to the people who take it, and an expert panel's ratings.
Skip logic and display logic
Rules that show a question only when earlier answers call for it.
Survey software
The question types, rules, scores and languages of the questionnaire editor.
Planning a study?
Start one now, or tell us what you want to measure.