Pilot study
A pilot study is a small run of a study before the main one. It checks that the study can be done as planned: that people can be found and will take part, that they understand what they are asked, that each part takes the time you expect, and that the data come out as the analysis needs them.
A pilot is not a small test of the hypothesis. Its aim is feasibility (Leon, Davis and Kraemer, 2011). It can also estimate how spread out the main outcome is, which the main study's sample size needs (Julious, 2005).
What a pilot checks
| Question | What to measure in the pilot |
|---|---|
| Can you recruit? | The people who join each week, and the share of those asked who take part |
| Do people qualify and consent? | The share who pass the screener, and the share who consent |
| Do they finish? | The share who start and finish, and the screen where those who stop leave |
| How long does it take? | The minutes for each part and for the whole study |
| Do they understand? | Questions left blank or answered in ways you did not expect, and what people tell you afterwards |
| Do the groups work? | Each group's size, and that each person got the condition they were assigned |
| Are the data right? | Every column the analysis needs is in the download, in the form it needs |
| How spread out is the outcome? | The standard deviation (SD) of the main outcome, for the main study's sample size |
Decide before the pilot what counts as a success, such as the share of people who must finish for the main study to go ahead. The CONSORT extension for pilot and feasibility trials asks you to report these criteria (Eldridge et al., 2016).
How many people a pilot needs
| Aim | People | Source |
|---|---|---|
| Estimate an outcome's SD for the main study's sample size | 12 in each group | Julious, 2005 |
| Estimates precise enough for the pilot's aims | 10 to 40 in each group, by the aim | Hertzog, 2008 |
Whitehead and colleagues (2016) chose the pilot's size so that the pilot and the main study together need the fewest people. For a main study with 90% power and a two-sided test at 0.05:
| The main study's expected d | Pilot people in each group |
|---|---|
| 0.1 or less | 75 |
| 0.2 | 25 |
| 0.5 | 15 |
| 0.8 | 10 |
A main study planned on a small pilot's SD is quite likely to have less power than planned. In Browne's (1995) Monte Carlo study, planning on a one-sided upper confidence limit of the SD, such as the 80% limit, gave the planned power in at least that share of studies.
Do not size the main study from the pilot's effect
A pilot with 12 people in each group that finds Cohen's d = 0.50 has a 95% interval from −0.32 to 1.31: the true effect could be nothing, or large. Planning on d = 0.50 gives 64 people in each group for 80% power. If the true d is 0.30, those 64 give 39% power, and 176 in each group were needed.
Kraemer and colleagues (2006) warned that when pilots size studies, the most likely outcomes are that studies worth doing are dropped and those that go ahead are short of power. Plan the main study on the smallest difference that would matter, or on effects from earlier studies.
Pilots in ConductExperience
- Preview runs the draft in your browser, as a participant sees it. Nothing is saved, and the camera is not used.
- Pilot it yourself starts a pilot on the study's latest version. If the draft has no version yet or has unpublished changes, it publishes the draft as a new version first.
- A pilot gives a pilot link and its QR code, or runs in lab sessions when the study's files stay on the lab computer. Pilot data is kept apart from your real data.
- Results shows the people in the study or the pilot people, as you choose.
- The download leaves pilot people out unless you tick Include pilot people.
- An owner can clear the pilot data. It removes the pilot sessions, and your real data is not touched.
Report a pilot
Report it as a pilot or feasibility study, with its aims, the criteria you set for going ahead and what you changed for the main study. The CONSORT 2010 extension to randomised pilot and feasibility trials lists what to report (Eldridge et al., 2016), and Thabane and colleagues (2010) wrote a tutorial on pilots.
Sources
- Browne, R. H. (1995). On the use of a pilot sample for sample size determination. Statistics in Medicine, 14(17), 1933–1940.
- Eldridge, S. M., et al. (2016). CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ, 355, i5239.
- Hertzog, M. A. (2008). Considerations in determining sample size for pilot studies. Research in Nursing & Health, 31(2), 180–191.
- Julious, S. A. (2005). Sample size of 12 per group rule of thumb for a pilot study. Pharmaceutical Statistics, 4(4), 287–291.
- Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry, 63(5), 484–489.
- Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626–629.
- Thabane, L., et al. (2010). A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology, 10, 1.
- Whitehead, A. L., Julious, S. A., Cooper, C. L., & Campbell, M. J. (2016). Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Statistical Methods in Medical Research, 25(3), 1057–1073.
Related
Sample size calculator
How many people the main study needs for the effect size you expect.
Effect size calculator
Cohen's d and Hedges' g with their intervals, from two groups' means and SDs.
Screener surveys
Questions that decide who fits, rules that turn away the rest, and quotas.
Study builder
Groups, sections and parts in fixed, shuffled or counterbalanced orders.
Planning a study?
Start one now, or tell us what you want to measure.