Tools for Researchers

Designing Experiments in the Presence of Interference

Introduction

The possibility of interference between individuals has traditionally been seen as the Achilles heel of randomized experiments, because contamination of the control group by spillover effects generates impact estimates that are internally invalid. Research designs and randomized control trials that fail to account for spillovers can produce biased estimates of intention-to-treat effects, while finding meaningful treatment effects but failing to observe deleterious spillovers can lead to misconstrued policy conclusions. In many contexts, a full understanding of the policy environment requires us to measure spillover and threshold effects that are not captured by (or, worse, are sources of bias in) standard experimental designs.

This software allows a researcher to explore the statistical power of experiments to identify estimands of treatment and spillover effects when there is interference between units. We focus on settings with partial interference, in which individuals are split into mutually exclusive clusters, such as villages or schools, and interference occurs between individuals within a cluster but not across clusters. We consider experiments in which treatment is allocated using a randomized saturation (RS) design, which is a two-stage randomization procedure in which first the share of individuals assigned to treatment within a cluster is randomized, and second, the individuals within each cluster are randomly assigned to treatment according to the realized cluster-level saturation from the first stage. RS designs can be used to identify a rich set of estimands, including the treatment effect and spillover effect on untreated individuals at specific saturations, slope effects measuring how spillover effects change with respect to treatment saturation, and pooled effects across multiple saturations.

For a given RS design, our software allows a user to calculate the minimum detectable effect (MDE) of these estimands, which is the smallest value of an estimand that it is possible to distinguish from zero. The software also calculates the optimal RS design for different researcher objectives. Given a set of estimands and a set of weights specified by the researcher, the software calculates the RS design that minimizes the weighted sum of the MDEs for the specified set of estimands. Our paper establishes that introducing variation into the treatment saturation of clusters impacts the power of different estimands. These optimal design calculations allow the researcher to precisely characterize this power trade-off.

Downloads

We have provided a GUI for ease of use. The video provides a tutorial on how to use it. We have also supplied Python, R, and MATLAB code. 

Citing the Software

A. Bohren, P. Staples, S. Baird, C. McIntosh, and B. Özler, (2016).  Power Calculation Software for Randomized Saturation Experiments, Version 1.0.  Available from http://pdel.ucsd.edu/solutions/index.html

Video Tutorial for the Application

Theoretical Underpinnings

Download the working paper