RCTs and Quasi-Experimental Designs

The simulation demonstrates four ideas: difference-in-means for an RCT, a randomization-inference p-value, a blocked estimator, and ITT/LATE under noncompliance.

import numpy as np
import pandas as pd

rng = np.random.default_rng(123)
n = 800
school = rng.integers(0, 8, size=n)
baseline = rng.normal(0, 1, size=n) + 0.25 * school

# Fixed potential outcomes.
y0 = 50 + 5 * baseline + rng.normal(0, 4, size=n)
tau = 3.0 + 1.2 * (baseline < 0)
y1 = y0 + tau
true_ate = np.mean(tau)

# Complete random assignment.
n1 = n // 2
d = np.zeros(n, dtype=int)
d[rng.choice(n, size=n1, replace=False)] = 1
y = d * y1 + (1 - d) * y0

def diff_in_means(outcome, treat):
    return outcome[treat == 1].mean() - outcome[treat == 0].mean()

tau_hat = diff_in_means(y, d)
se_neyman = np.sqrt(y[d == 1].var(ddof=1) / d.sum() + y[d == 0].var(ddof=1) / (n - d.sum()))

# Randomization inference under the sharp null of no effect.
observed_t = diff_in_means(y, d)
null_stats = []
for _ in range(5000):
    d_perm = np.zeros(n, dtype=int)
    d_perm[rng.choice(n, size=n1, replace=False)] = 1
    null_stats.append(diff_in_means(y, d_perm))
p_value = np.mean(np.abs(null_stats) >= abs(observed_t))

# Blocked estimate: compare within school, then aggregate.
df = pd.DataFrame({"Y": y, "D": d, "school": school})
blocked = (
    df.groupby("school")
      .apply(lambda g: pd.Series({
          "n": len(g),
          "tau_b": diff_in_means(g["Y"].to_numpy(), g["D"].to_numpy())
      }))
      .reset_index()
)
tau_blocked = np.average(blocked["tau_b"], weights=blocked["n"])

# Noncompliance: randomized offer Z, actual take-up D.
z = np.zeros(n, dtype=int)
z[rng.choice(n, size=n1, replace=False)] = 1
complier = rng.binomial(1, 0.72, size=n)
always_taker = rng.binomial(1, 0.08, size=n)
d_actual = np.maximum(always_taker, z * complier)
y_nc = y0 + d_actual * tau + rng.normal(0, 2, size=n)

itt_y = diff_in_means(y_nc, z)
itt_d = diff_in_means(d_actual, z)
late = itt_y / itt_d

print({
    "true_ate": round(true_ate, 3),
    "rct_tau_hat": round(tau_hat, 3),
    "neyman_se": round(se_neyman, 3),
    "randomization_p": round(p_value, 4),
    "blocked_tau_hat": round(tau_blocked, 3),
    "itt_y": round(itt_y, 3),
    "itt_d": round(itt_d, 3),
    "late_wald": round(late, 3),
})

• Question: does a summer data bootcamp improve later project scores? If applicants exceed seats, a lottery offer can serve as randomized encouragement.
• The primary analysis reports ITT: whether being offered a seat raises average outcomes. This matches the policy effect of offering seats or invitations.
• If some offered students do not attend and some non-offered students find another route in, actual participation D is not fully randomized. With defensible exclusion and monotonicity, offer Z can estimate complier LATE.
• Without a lottery, assignment may come from a cutoff, regional pilot, grade cohort, or policy timing. That becomes quasi-experimental, and the rule itself must justify local as-if random variation.
• A credible report includes the experiment flow, preregistration or sample flow, baseline balance, attrition, noncompliance, spillovers, multiple testing, randomization inference or clustered standard errors, and all deviations from the original design.

References

• Fisher (1935), The Design of Experimentshttps://archive.org/details/designofexperime00fish
• Neyman (1990 translation), On the Application of Probability Theory to Agricultural Experimentshttps://doi.org/10.1214/ss/1177012031
• Duflo, Glennerster, and Kremer (2007), Using Randomization in Development Economics Researchhttps://doi.org/10.1016/S1573-4471(07)04061-2
• Gerber and Green, Field Experimentshttps://wwnorton.com/books/Field-Experiments/