Experimental Methods

A 4-Day Remote Seminar Taught by
Henry May, Ph.D.

Read reviews of this course

To see a sample of the course materials, click here.


This intermediate course on experimental design and analysis will focus on causal inference in randomized field trials. Topics include randomization-based causal inference, design of multilevel experiments (including cluster and multisite designs), statistical power, multiple comparisons, implementation fidelity, and techniques for addressing imperfections in real-world experiments (e.g., attrition, non-compliance).

The primary goal of any randomized experiment is to generate trustworthy inferences about the causal effects of a treatment or intervention. While randomized experiments have been a gold-standard for causal inference in medicine since the 1946 randomized clinical trial (RCT) of streptomycin, randomization-based experiments have only recently become the preferred method for studying the effects of social interventions.

For example, in the field of education, the past two decades have seen a dramatic shift toward understanding “what works” through randomized field trials as espoused by federal legislation and research grant programs. This seminar covers key aspects of designing and conducting experiments that meet the requirements of many federal and foundation grants programs focused on intervention research.

Starting July 13, we are offering this seminar as a 4-day synchronous*, remote workshop. Each day will consist of a 3-hour live lecture held via the free video-conferencing software Zoom. You are encouraged to join the lecture live, but will have the opportunity to view the recorded session later in the day if you are unable to attend at the scheduled time.

Each lecture session will conclude with a hands-on exercise reviewing the content covered, to be completed on your own. An additional lab session will be held Tuesday and Thursday afternoons, where you can review the exercise results with the instructor and ask any questions.

*We understand that scheduling is difficult during this unpredictable time. If you prefer, you may take all or part of the course asynchronously. The video recordings will be made available within 24 hours of each session and will be accessible for two weeks after the seminar, meaning that you will get all of the class content and discussions even if you cannot participate synchronously.

Closed captioning is available for all live and recorded sessions.


MORE DETAILS ABOUT THE COURSE CONTENT

Examples used in the course demonstrate application of modern techniques for randomized experiments in real-world settings for social interventions (e.g., in schools, communities), marketing and sales interventions (e.g., in retail stores), and medical education/training (e.g., in hospitals). Additionally, the seminar covers key issues for experimental research subjected to the standards set by the US Department of Education’s What Works Clearinghouse and other policies from the National Institutes of Health and www.ClinicalTrials.gov.


COMPUTING

Because this is a hands-on course, you are strongly encouraged to use your own laptop or desktop computer (Mac or Windows) with one of the following packages installed:

  • SPSS Statistics (version 23 or later)
  • SAS (release 9.2 or later)
  • R (v3.6 or later) installed.

You can choose which statistics package you want to use while working through the course material.

You should have good familiarity with the basics of ordinary least squares regression as well as the use of SPSS, SAS, or R, including opening and executing data files and programs. You are also encouraged to have your own data available to apply what you’ve learned.

If you’d like to use R for this course but don’t yet have much experience with that package, here are some excellent on-line resources for building your R skills.


WHO SHOULD Register? 

This course will be helpful for researchers in any field—including psychology, sociology, education, business, human development, political science, public health, communication—and others who want to learn how to plan, design, and analyze data from randomized experiments using readily-available software packages. You should have a basic working knowledge of the principles and practice of multiple regression and elementary statistical inference. Experience with multilevel models (e.g., mixed effects and random effects models, also known as hierarchical linear models, or HLM) will be especially helpful, but is not required. No knowledge of matrix algebra is required or assumed.


Seminar Outline

Day 1: Experimental Design and Causal Inference

1. Theoretical Foundations of Causal Inference in Experiments
     A. Campbell’s Experimental Framework
     B. Rubin’s Causal Model (Potential Outcomes)
2. RCT Design Options
     A. A Simple Person-Randomized Design
     B. Blocked Randomization
     C. Cluster-Randomized Trials (2-level designs)
     D. Multisite Trials (2-level designs)
     E. Multisite Cluster Trials (3-level designs)
     F. Matched-Pair Designs
3. Measurement of Covariates and Outcomes
     A. Pre-Treatment Measures
     B. Confirming Baseline Balance
     C. Post-Treatment Outcome Measures

Day 2: Analyzing Data from Experiments

4. Documenting Treatment versus Control Conditions
     A. Treatment Implementation Fidelity
     B. Documenting the Counterfactual
     C. Control Group Contamination
5. Statistical Models for Impacts on Continuous Outcomes
     A. Single-Level Analyses
     B. Mixed-Effects Models for Multilevel Designs
     C. Fixed-Effects Models for Multisite Designs
     D. Other Alternatives (e.g., GEE, clustered standard errors)

 Day 3: Categorical Outcomes, Sample Size/Power Analysis

6. Statistical Models for Impacts on Categorical Outcomes
     A. Single-Level Analyses
     B. Mixed-Effects Models for Multilevel Designs
     C. Fixed-Effects Models for Multisite Designs
     D. Other Alternatives (e.g., GEE, clustered standard errors)
7. Power Analyses
     A. Single-Level RCTs
     B. Multilevel RCTs
     C. Power for Complex Designs (i.e., Monte Carlo Simulation) 

Day 4: Techniques for Handling Real-World Complexities

8. Beyond Basic Impact Analyses
     A. Multiple Comparisons (i.e., Familywise Error vs. False Discovery Rate)
     B. Local Average Treatment Effects
            a. Intent-to-Treat vs. Treatment-on-the-Treated Effects
            b. Handling no-shows and crossovers
     C. Attrition, Non-response, & Missing Data
9. Reporting in RCTs (alternative effect sizes, CONSORT diagrams, WWC standards)
10. Demonstration / Activity
     A. Calculating CACE
     B. Implement B-H control of FDR


REVIEWS OF Experimental Methods

“This is a wonderful introduction course to experimental methods. Dr. May is a very effective instructor. He’s knowledgeable and clear, and the material he covers is thorough, with a good balance of comprehensiveness and depth. Highly recommended.”
  Bonnie Wu, Wayne State University

“The instructor provides clear explanation and is easy to follow.”
  Anonymous

“I felt comfortable asking any question and the responses were always clear and detailed. Dr. May is professional and has impressive knowledge and expertise in the topic. Any question, at any level of difficulty, was answered so well. The real-world and real-research examples provided by Dr. May were also extremely useful. I could make parallels with my own planned research design even though I am from a different discipline. Finally, I liked seeing Dr. May code and de-bug in R in real-time. Even the biggest minds have bugs in code sometimes. This was very interactive and I liked this (especially as a beginner with R).”
  Anonymous