Testing People, and the Experimenter Effect Problem

Wiseman has also run direct tests of individual claimants, most publicly the 2004 examination of Natasha Demkina, a young Russian woman who said she could see inside bodies and diagnose conditions. The test, designed with Ray Hyman and the physician Andrew Skolnick and filmed for television, set a threshold in advance: she was asked to match seven medical conditions to seven volunteers and needed five correct to proceed to a fuller study. She got four. Both sides then disputed what that meant, since four of seven is well above the chance expectation for the task while below the criterion everyone had accepted beforehand, and the episode is a useful illustration of why pre-specified thresholds are difficult to hold to once a result arrives.

A more subtle strand of his work concerns the experimenter effect: in a series of joint studies with the parapsychologist Marilyn Schlitz, the same protocol run in the same laboratory produced positive results when she was the experimenter and null results when he was. Neither of them has offered a settled explanation, and the finding cuts both ways, since it could indicate subtle procedural differences or something about expectation that neither side's theory handles well.