Evaluating factorial experimental designs with imperfect compliance: Evidence from a durable early childhood intervention

Factorial designs are popular because they allow researchers to evaluate complementarity between program components. We show that when compliance is imperfect, standard tests of complementarity in factorial designs can produce false-positive and false-negative results. We illustrate this by evaluating the long-term impact of an early childhood intervention that combined a Parent Academy and a high-quality preschool. The combined program positively affected grades ten years after completion, contrary to the Parent Academy-only and preschool-only programs. We develop a compliance-robust test of the complementarity of programs and show that the long-term effects of the combined programs are consistent with the presence of complementarity. The methods developed apply to other factorial designs.