Skip to content
ConductExperience
Menu

Cognitive interviewing

Cognitive interviewing tests a questionnaire before a study uses it. People answer the draft questions while an interviewer finds out how they understood each one and how they chose their answer.

It helps show whether each question gets the information you meant it to, and it is one of the main ways to find and fix problems with survey questions (Beatty and Willis, 2007).

What can go wrong when a person answers

StepWhat can go wrongA probe for it
Understanding the questionA word or phrase means something else to the personWhat does "in the past week" mean to you?
RememberingThe person cannot recall what the question asks forHow did you work out that number?
JudgingThe person guesses, adds up loosely or answers to look goodHow sure are you of that answer?
AnsweringThe person's answer does not fit the optionsWas it easy or hard to choose an answer? Why?

The four steps follow the model of Tourangeau, Rips and Rasinski (2000), who call them comprehension, retrieval, judgment and response.

Think-aloud or probing

MethodHow it runsTrade-off
Think-aloudThe person says what they think while they answer each question, and the interviewer listensLittle steering from the interviewer, but some people find it hard to keep talking
Concurrent probingThe interviewer asks planned or spontaneous probes after each questionFocused answers, but the probes can change how the next questions are read
Retrospective probingThe person answers the whole questionnaire, then the interviewer goes back over itThe questionnaire runs as it will in the study, but people may not remember why they answered as they did

Beatty and Willis (2007) review these ways of working and their apparent advantages, and note where standards for best practice are not clearly defined, so take the trade-offs as a guide, not a rule.

Rounds

  • Run the interviews in small rounds. After each round, change the questions that caused trouble, and test the new wording in the next round.
  • Choose people like those the study will ask, including those most likely to struggle with the questions.
  • Write down, for each question, what went wrong and what you changed: the record shows why the final wording is what it is.

Who to interview, how many interviews to run and how to choose the probes are decisions each study makes, with no fixed standard (Beatty and Willis, 2007).

Web probing in ConductExperience

Web probing asks the probes in the questionnaire itself, so many more people answer them than an interviewer can see:

  • After a question you are testing, add a long text question that asks how the person read it, with up to 2,000 characters for the answer.
  • Rules show a probe only when an earlier answer calls for it.
  • Piped text puts the person's earlier answer into the probe's words: "You chose Often. What did you count as often?"
  • Results lists each written answer with the person's study code. Only people who can download the data see them.

Sources

  • Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311.
  • Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press.
  • Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.

Related

  • Readability test

    Flesch Reading Ease, Flesch-Kincaid grade, SMOG and the fog index of a consent form or instructions.

  • Face validity

    Whether a questionnaire looks right to the people who take it, and an expert panel's ratings.

  • Skip logic and display logic

    Rules that show a question only when earlier answers call for it.

  • Survey software

    The question types, rules, scores and languages of the questionnaire editor.

Planning a study?

Start one now, or tell us what you want to measure.