Skip to content
ConductExperience
Menu

Pilot study

A pilot study is a small run of a study before the main one. It checks that the study can be done as planned: that people can be found and will take part, that they understand what they are asked, that each part takes the time you expect, and that the data come out as the analysis needs them.

A pilot is not a small test of the hypothesis. Its aim is feasibility (Leon, Davis and Kraemer, 2011). It can also estimate how spread out the main outcome is, which the main study's sample size needs (Julious, 2005).

What a pilot checks

QuestionWhat to measure in the pilot
Can you recruit?The people who join each week, and the share of those asked who take part
Do people qualify and consent?The share who pass the screener, and the share who consent
Do they finish?The share who start and finish, and the screen where those who stop leave
How long does it take?The minutes for each part and for the whole study
Do they understand?Questions left blank or answered in ways you did not expect, and what people tell you afterwards
Do the groups work?Each group's size, and that each person got the condition they were assigned
Are the data right?Every column the analysis needs is in the download, in the form it needs
How spread out is the outcome?The standard deviation (SD) of the main outcome, for the main study's sample size

Decide before the pilot what counts as a success, such as the share of people who must finish for the main study to go ahead. The CONSORT extension for pilot and feasibility trials asks you to report these criteria (Eldridge et al., 2016).

How many people a pilot needs

AimPeopleSource
Estimate an outcome's SD for the main study's sample size12 in each groupJulious, 2005
Estimates precise enough for the pilot's aims10 to 40 in each group, by the aimHertzog, 2008

Whitehead and colleagues (2016) chose the pilot's size so that the pilot and the main study together need the fewest people. For a main study with 90% power and a two-sided test at 0.05:

The main study's expected dPilot people in each group
0.1 or less75
0.225
0.515
0.810

A main study planned on a small pilot's SD is quite likely to have less power than planned. In Browne's (1995) Monte Carlo study, planning on a one-sided upper confidence limit of the SD, such as the 80% limit, gave the planned power in at least that share of studies.

Do not size the main study from the pilot's effect

A pilot with 12 people in each group that finds Cohen's d = 0.50 has a 95% interval from −0.32 to 1.31: the true effect could be nothing, or large. Planning on d = 0.50 gives 64 people in each group for 80% power. If the true d is 0.30, those 64 give 39% power, and 176 in each group were needed.

Kraemer and colleagues (2006) warned that when pilots size studies, the most likely outcomes are that studies worth doing are dropped and those that go ahead are short of power. Plan the main study on the smallest difference that would matter, or on effects from earlier studies.

Pilots in ConductExperience

  • Preview runs the draft in your browser, as a participant sees it. Nothing is saved, and the camera is not used.
  • Pilot it yourself starts a pilot on the study's latest version. If the draft has no version yet or has unpublished changes, it publishes the draft as a new version first.
  • A pilot gives a pilot link and its QR code, or runs in lab sessions when the study's files stay on the lab computer. Pilot data is kept apart from your real data.
  • Results shows the people in the study or the pilot people, as you choose.
  • The download leaves pilot people out unless you tick Include pilot people.
  • An owner can clear the pilot data. It removes the pilot sessions, and your real data is not touched.

Report a pilot

Report it as a pilot or feasibility study, with its aims, the criteria you set for going ahead and what you changed for the main study. The CONSORT 2010 extension to randomised pilot and feasibility trials lists what to report (Eldridge et al., 2016), and Thabane and colleagues (2010) wrote a tutorial on pilots.

Sources

  • Browne, R. H. (1995). On the use of a pilot sample for sample size determination. Statistics in Medicine, 14(17), 1933–1940.
  • Eldridge, S. M., et al. (2016). CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ, 355, i5239.
  • Hertzog, M. A. (2008). Considerations in determining sample size for pilot studies. Research in Nursing & Health, 31(2), 180–191.
  • Julious, S. A. (2005). Sample size of 12 per group rule of thumb for a pilot study. Pharmaceutical Statistics, 4(4), 287–291.
  • Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry, 63(5), 484–489.
  • Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626–629.
  • Thabane, L., et al. (2010). A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology, 10, 1.
  • Whitehead, A. L., Julious, S. A., Cooper, C. L., & Campbell, M. J. (2016). Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Statistical Methods in Medical Research, 25(3), 1057–1073.

Related

Planning a study?

Start one now, or tell us what you want to measure.