Experimental Methods

A 3-Day Remote Seminar Taught by
Henry May, Ph.D.

To see a sample of the course materials, click here.


This intermediate course on experimental design and analysis will focus on causal inference in randomized field trials. Topics include randomization-based causal inference, design of multilevel experiments (including cluster and multisite designs), statistical power, multiple comparisons, implementation fidelity, and techniques for addressing imperfections in real-world experiments (e.g., attrition, non-compliance).

The primary goal of any randomized experiment is to generate trustworthy inferences about the causal effects of a treatment or intervention. While randomized experiments have been a gold-standard for causal inference in medicine since the 1946 randomized clinical trial (RCT) of streptomycin, randomization-based experiments have only recently become the preferred method for studying the effects of social interventions.

For example, in the field of education, the past two decades have seen a dramatic shift toward understanding “what works” through randomized field trials as espoused by federal legislation and research grant programs. This seminar covers key aspects of designing and conducting experiments that meet the requirements of many federal and foundation grants programs focused on intervention research.

Starting January 14, we are offering this seminar as a 3-day synchronous*, remote workshop for the first time. Each day will consist of a 4-hour live lecture held via the free video-conferencing software Zoom. You are encouraged to join the lecture live, but will have the opportunity to view the recorded session later in the day if you are unable to attend at the scheduled time.

Each lecture session will conclude with a hands-on exercise reviewing the content covered, to be completed on your own. An additional session will be held Thursday and Friday afternoons as an “office hour”, where you can review the exercise results with the instructor and ask any questions.

*We understand that scheduling is difficult during this unpredictable time. If you prefer, you may take all or part of the course asynchronously. The video recordings will be made available within 24 hours of each session and will be accessible for two weeks after the seminar, meaning that you will get all of the class content and discussions even if you cannot participate synchronously. 


MORE DETAILS ABOUT THE COURSE CONTENT

Examples used in the course demonstrate application of modern techniques for randomized experiments in real-world settings for social interventions (e.g., in schools, communities), marketing and sales interventions (e.g., in retail stores), and medical education/training (e.g., in hospitals). Additionally, the seminar covers key issues for experimental research subjected to the standards set by the US Department of Education’s What Works Clearinghouse and other policies from the National Institutes of Health and www.ClinicalTrials.gov.


COMPUTING

This remote seminar is held via Zoom, a free video conferencing application. Instructions for joining a session via Zoom are available here. Before the seminar begins, you will receive an email with the meeting code and password you must use to join.  

Because this is a hands-on course, you are strongly encouraged to use your own laptop or desktop computer (Mac or Windows) with one of the following packages installed:

  • SPSS Statistics (version 23 or later)
  • SAS (release 9.2 or later)
  • R (v3.6 or later) installed.

You can choose which statistics package you want to use while working through the course material.

You should have good familiarity with the basics of ordinary least squares regression as well as the use of SPSS, SAS, or R, including opening and executing data files and programs. You are also encouraged to have your own data available to apply what you’ve learned.

If you’d like to take this course but are concerned that you don’t know enough R, there are excellent on-line resources for learning the basics. Here are our recommendations.


WHO SHOULD Register?

This course will be helpful for researchers in any field—including psychology, sociology, education, business, human development, political science, public health, communication—and others who want to learn how to plan, design, and analyze data from randomized experiments using readily-available software packages. You should have a basic working knowledge of the principles and practice of multiple regression and elementary statistical inference. Experience with multilevel models (e.g., mixed effects and random effects models, also known as hierarchical linear models, or HLM) will be especially helpful, but is not required. No knowledge of matrix algebra is required or assumed.


SEMINAR OUTLINE

1. Theoretical Foundations of Causal Inference in Experiments
     a. Campbell’s Experimental Framework
     b. Rubin’s Causal Model (Potential Outcomes)
2. RCT Design Options
     a. A Simple Person-Randomized Design
     b. Blocked Randomization
     c. Cluster-Randomized Trials (2-level designs)
     d. Multisite Trials (2-level designs)
     e. Multisite Cluster Trials (3-level designs)
     f. Matched-Pair Designs
3. Measurement of Covariates and Outcomes
     a. Pre-Treatment Measures 
     b. Confirming Baseline Balance
     c. Post-Treatment Outcome Measures
4. Statistical Models for Impact Analyses
     a. Single-Level Analyses
     b. Mixed-Effects Models for Multilevel Designs
     c. Fixed-Effects Models for Multisite Designs
     d. Other Alternatives (e.g., GEE, clustered standard errors)
5. Documenting Treatment versus Control Conditions
     a. Treatment Implementation Fidelity
     a. Documenting the Counterfactual 
     a. Control Group Contamination
6. Power Analyses
     a. Single-Level RCTs
     b. Multilevel RCTs
     c. Power for Complex Designs (i.e., Monte Carlo Simulation)
7. Beyond Basic Impact Analyses
     a. Multiple Comparisons (i.e., Familywise Error vs. False Discovery Rate)
     b. Local Average Treatment Effects
            i. Intent-to-Treat vs. Treatment-on-the-Treated Effects
            ii. Handling no-shows and crossovers
     c. Attrition, Non-response, & Missing Data
8. Reporting in RCTs (alternative effect sizes, CONSORT diagrams, WWC standards)