Why the Composite Score Is the Problem

The statistical objection to this trial is standard and is worth spelling out because it applies far beyond prayer research. If a study measures a couple of dozen outcomes and tests each at the conventional threshold, roughly one in twenty will separate the groups by chance alone even if nothing whatever is going on. Reporting only the ones that separated, or folding them into a composite constructed after the results are known, hides that arithmetic. The remedy is to name a single primary outcome before the data are collected and to correct the threshold for everything else examined.

Byrd's trial did not do this, and the outcomes that reached significance were not ones a stated prior hypothesis had identified. William Harris and colleagues attempted a stricter version in 1999 with a larger sample and reported a small positive effect on a similar composite, drawing similar objections. The question was then settled as far as this literature can settle anything by the STEP trial in 2006, which pre-specified its outcome, enrolled roughly four and a half times as many patients, and found nothing. A Cochrane review has since concluded the evidence does not warrant further trials.