Experimental Methods - Online Course
A 4-Day Livestream Seminar Taught by
Henry MayTuesday, July 23 –
Friday, July 26, 2024
10:30am-12:30pm (convert to your local time)
1:30pm-3:00pm
This intermediate course on experimental design and analysis will focus on causal inference in randomized field trials. Topics include randomization-based causal inference, design of multilevel experiments (including cluster and multisite designs), statistical power, multiple comparisons, implementation fidelity, and techniques for addressing imperfections in real-world experiments (e.g., attrition, non-compliance).
The primary goal of any randomized experiment is to generate trustworthy inferences about the causal effects of a treatment or intervention. While randomized experiments have been a gold-standard for causal inference in medicine since the 1946 randomized clinical trial (RCT) of streptomycin, randomization-based experiments have only recently become the preferred method for studying the effects of social interventions.
For example, in the field of education, the past two decades have seen a dramatic shift toward understanding “what works” through randomized field trials as espoused by federal legislation and research grant programs. This seminar covers key aspects of designing and conducting experiments that meet the requirements of many federal and foundation grants programs focused on intervention research.
Starting July 23, we are offering this seminar as a 4-day synchronous*, livestream workshop held via the free video-conferencing software Zoom. Each day will consist of two lecture sessions which include hands-on exercises, separated by a 1-hour break. You are encouraged to join the lecture live, but will have the opportunity to view the recorded session later in the day if you are unable to attend at the scheduled time.
*We understand that finding time to participate in livestream courses can be difficult. If you prefer, you may take all or part of the course asynchronously. The video recordings will be made available within 24 hours of each session and will be accessible for four weeks after the seminar, meaning that you will get all of the class content and discussions even if you cannot participate synchronously.
Closed captioning is available for all live and recorded sessions. Live captions can be translated to a variety of languages including Spanish, Korean, and Italian. For more information, click here.
More details about the course content
Examples used in the course demonstrate application of modern techniques for randomized experiments in real-world settings for social interventions (e.g., in schools, communities), marketing and sales interventions (e.g., in retail stores), and medical education/training (e.g., in hospitals). Additionally, the seminar covers key issues for experimental research subjected to the standards set by the US Department of Education’s What Works Clearinghouse and other policies from the National Institutes of Health and www.ClinicalTrials.gov.
Examples used in the course demonstrate application of modern techniques for randomized experiments in real-world settings for social interventions (e.g., in schools, communities), marketing and sales interventions (e.g., in retail stores), and medical education/training (e.g., in hospitals). Additionally, the seminar covers key issues for experimental research subjected to the standards set by the US Department of Education’s What Works Clearinghouse and other policies from the National Institutes of Health and www.ClinicalTrials.gov.
Computing
Because this is a hands-on course, you are strongly encouraged to use your own laptop or desktop computer (Mac or Windows) with one of the following packages installed:
- R (v3.6 or later)
- SPSS Statistics (version 23 or later)
- SAS (release 9.2 or later)
Note that analyses using SAS will not be demonstrated during the live seminar, but SAS code that replicates the results demonstrated using R and SPSS will be provided.
You should have good familiarity with the basics of ordinary least squares regression as well as the use of R, SPSS, or SAS, including opening and executing data files and programs. You are also encouraged to have your own data available to apply what you’ve learned.
If you’d like to use R for this course but don’t yet have much experience with that package, here are some excellent on-line resources for building your R skills.
There is now a free version of SAS, called SAS OnDemand for Academics, that is available to anyone.
Because this is a hands-on course, you are strongly encouraged to use your own laptop or desktop computer (Mac or Windows) with one of the following packages installed:
- R (v3.6 or later)
- SPSS Statistics (version 23 or later)
- SAS (release 9.2 or later)
Note that analyses using SAS will not be demonstrated during the live seminar, but SAS code that replicates the results demonstrated using R and SPSS will be provided.
You should have good familiarity with the basics of ordinary least squares regression as well as the use of R, SPSS, or SAS, including opening and executing data files and programs. You are also encouraged to have your own data available to apply what you’ve learned.
If you’d like to use R for this course but don’t yet have much experience with that package, here are some excellent on-line resources for building your R skills.
There is now a free version of SAS, called SAS OnDemand for Academics, that is available to anyone.
Who should register?
This course will be helpful for researchers in any field—including psychology, sociology, education, business, human development, political science, public health, and communications—and others who want to learn how to plan, design, and analyze data from randomized experiments using readily-available software packages. You should have a basic working knowledge of the principles and practice of multiple regression and elementary statistical inference. Experience with multilevel models (e.g., mixed effects and random effects models, also known as hierarchical linear models, or HLM) will be especially helpful, but is not required. No knowledge of matrix algebra is required or assumed.
This course will be helpful for researchers in any field—including psychology, sociology, education, business, human development, political science, public health, and communications—and others who want to learn how to plan, design, and analyze data from randomized experiments using readily-available software packages. You should have a basic working knowledge of the principles and practice of multiple regression and elementary statistical inference. Experience with multilevel models (e.g., mixed effects and random effects models, also known as hierarchical linear models, or HLM) will be especially helpful, but is not required. No knowledge of matrix algebra is required or assumed.
Seminar outline
- Theoretical foundations of causal inference in experiments
- Campbell’s experimental framework
- Rubin’s causal model (potential outcomes)
- RCT design options
- A simple person-randomized design
- Blocked randomization
- Cluster-randomized trials (2-level designs)
- Multisite trials (2-level designs)
- Multisite cluster trials (3-level designs)
- Matched-pair designs
- Measurement of covariates and outcomes
- Pre-treatment measures
- Confirming baseline balance
- Post-treatment outcome measures
- Documenting treatment versus control conditions
- Treatment implementation fidelity
- Documenting the counterfactual
- Control group contamination
- Statistical models for impacts on continuous outcomes
- Single-level analyses
- Mixed-effects models for multilevel designs
- Fixed-effects models for multisite designs
- Other alternatives (e.g., GEE, clustered standard errors)
- Statistical models for impacts on categorical outcomes
- Single-level analyses
- Mixed-effects models for multilevel designs
- Fixed-effects models for multisite designs
- Other alternatives (e.g., GEE, clustered standard errors)
- Power analyses
- Single-level RCTs
- Multilevel RCTs
- Power for complex designs (i.e., Monte Carlo simulation)
- Beyond basic impact analyses
- Multiple comparisons (i.e., familywise error vs. false discovery rate)
- Local average treatment effects
- Intent-to-treat vs. treatment-on-the-treated effects
- Handling no-shows and crossovers
- Attrition, non-response, & missing data
- Reporting in RCTs (alternative effect sizes, CONSORT diagrams, WWC standards)
- Demonstration / activity
- Calculating CACE
- Implement B-H control of FDR
- Theoretical foundations of causal inference in experiments
- Campbell’s experimental framework
- Rubin’s causal model (potential outcomes)
- RCT design options
- A simple person-randomized design
- Blocked randomization
- Cluster-randomized trials (2-level designs)
- Multisite trials (2-level designs)
- Multisite cluster trials (3-level designs)
- Matched-pair designs
- Measurement of covariates and outcomes
- Pre-treatment measures
- Confirming baseline balance
- Post-treatment outcome measures
- Documenting treatment versus control conditions
- Treatment implementation fidelity
- Documenting the counterfactual
- Control group contamination
- Statistical models for impacts on continuous outcomes
- Single-level analyses
- Mixed-effects models for multilevel designs
- Fixed-effects models for multisite designs
- Other alternatives (e.g., GEE, clustered standard errors)
- Statistical models for impacts on categorical outcomes
- Single-level analyses
- Mixed-effects models for multilevel designs
- Fixed-effects models for multisite designs
- Other alternatives (e.g., GEE, clustered standard errors)
- Power analyses
- Single-level RCTs
- Multilevel RCTs
- Power for complex designs (i.e., Monte Carlo simulation)
- Beyond basic impact analyses
- Multiple comparisons (i.e., familywise error vs. false discovery rate)
- Local average treatment effects
- Intent-to-treat vs. treatment-on-the-treated effects
- Handling no-shows and crossovers
- Attrition, non-response, & missing data
- Reporting in RCTs (alternative effect sizes, CONSORT diagrams, WWC standards)
- Demonstration / activity
- Calculating CACE
- Implement B-H control of FDR
Payment information
The fee of $995 includes all course materials.
PayPal and all major credit cards are accepted.
Our Tax ID number is 26-4576270.
The fee of $995 includes all course materials.
PayPal and all major credit cards are accepted.
Our Tax ID number is 26-4576270.