In my November and December posts, I extolled the virtues of SEM for estimating dynamic panel models. By combining fixed effects with lagged values of the predictor variables, I argued that this approach offers the best option for making causal inferences with non-experimental panel data. It controls for all time-invariant variables, whether observed or not, and it allows for the possibility of reciprocal causation. Simulation evidence strongly favors SEM over the more popular Arellano-Bond method for estimating these kinds of models.
Despite the potential for this method, I recently learned that it’s vulnerable to a very troubling kind of bias when the lag structure is misspecified. In the latest issue of Sociological Methods and Research, Stephen Vaisey and Andrew Miles showed by both simulation and formal proof that a positive contemporaneous effect will often show up as a negative effect when estimating a fixed effects model with a predictor that is lagged by one time unit. They concluded that, for most social science applications, “artifactual negative ‘effects’ will likely be the rule rather than the exception.”
Vaisey and Miles investigated this problem only for the case of three periods of data, no lagged effect of the dependent variable y on itself, and no true effect of y on x. In that case, maximum likelihood reduces to OLS regression using difference scores: y3–y2 on x2–x1. They showed that the coefficient for x2–x1 has an expected value that is exactly -.5 times the true coefficient.
LEARN MORE IN A SEMINAR WITH PAUL ALLISON
My own simulations suggest that a sign reversal can also happen with four or more periods and a lagged dependent variable. And the effect of one variable on the other doesn’t have to be exactly contemporaneous. The reversal of sign can also occur if the correct lag is one week, but the estimated model specifies a lag of one year. Note that this artifact does not arise with random effects models. It’s specific to fixed effects models with lagged predictors. That should not be interpreted as an endorsement of random effects models, however, because they are much more prone to bias from omitted variables.
As noted by Vaisey and Miles, a 2011 article in the Journal of Quantitative Criminology may exemplify the problem of misspecified lags. Following my advice, Ousey, Wilcox and Fisher used the fixed effects SEM method to examine the relationship between victimization and offending. Numerous studies have found a positive, cross-sectional relationship between these variables: people who report being victims of crimes are also more likely to commit crimes. But Ousey et al. found negative effects of each variable on the other. Respondents who reported higher levels of offending in year t had lower levels of victimization in year t+1, after adjusting for fixed effects. And respondents with higher levels of victimization in year t had lower levels of offending in year t+1.
This surprising result could be real. But it could also occur if there is a positive effect of victimization on offending that is almost instantaneous rather than lagged by one year. And, finally, it could also occur if there is a positive, instantaneous effect of offending on victimization.
What can be done about this problem? Well, one implication is that more thought should go into the design of panel surveys. If you expect that changes in x will produce changes in y a month later, then collecting monthly data would be much better than collecting annual data. This could have the added advantage of reducing the total time for data collection, although it might also increase certain kinds of response bias.
What if your data have already been collected? Here’s a tentative recommendation that worked well in a few simulations. As a robustness check, estimate models that include both contemporaneous and lagged predictors. If a one-year lag is the correct specification, then the contemporaneous effect should be small and not statistically significant. If, on the other hand, the contemporaneous effect is large and significant, it should raise serious doubts about validity of the method and the kinds of conclusions that can be drawn. It may be that the data are simply not suitable for separating the effect of x on y from the effect of y on x.
I tried this strategy on a subset of the data used by Ousey et al. to study victimization and offending. When both contemporaneous and lagged predictors were included, I found a strong positive effect of victimization on offending in the same year. The one-year lagged effect was negative but small and non-significant. The same thing happened in the reverse direction. Offending had a strong positive effect on victimization in the same year, but the lagged effect was negative and not significant. My take: these data don’t allow one to draw any firm conclusions about whether victimization affects offending or offending affects victimization. They certainly don’t provide a basis for claiming negative effects of each variable on the other.
Clearly this is a problem that needs a great deal more study. There is a substantial econometric literature on determining the number of lags needed for autoregressive models but, as far as I know, Vaisey and Miles are the first to identify this particular phenomenon.
By the way, Steve Vaisey teaches a highly-rated course for Statistical Horizons called Treatment Effects Analysis.
Comments
In LINEAR PANEL ANALYSIS: MODELS OF QUANTITATIVE CHANGE, now several decades old,(Academic Press, out of print), Ronald Kessler and I devoted some space of temporal misspecification. In a model that did NOT include fixed effects, we demonstrated that a wrong assum0tion about the lag could lead to a sign reversal. I don’t believe there is anything about adding fixed effects to the model that would change this. In researchl one may well want to include them, but for demonstrating the point, they are not needed. Our book recommends including both lagged and coteemporaneous effects when there is reason to expect them. It also briefly discusses continuous time models.
Thanks, David, for the heads up. Your book is on my shelf, but haven’t looked at it in years. I’ll check it out.
I wasn’t either until coming across this: Voelkle, M. C., Oud, J. H. L., Davidov, E., & Schmidt, P. (2012). An SEM approach to continuous time modeling of panel data: relating authoritarianism and anomia. Psychological Methods, 17, 176-192. doi: 10.1037/a0027543
Hi Paul,
Thank you for this post and the alert to the upcoming SMR paper. It seems that there might be multiple ways to interpret the findings of Vaisey & Miles as well as others who find interesting and strange differences in contemporaneous and lagged effects with panel data models. From one perspective, such as Granger’s (1969, 1980), any contemporaneous and therefore instantaneous effects are actually either (a) misspecified lagged effects that are finer-grained than the data allow investigating or are due to oscillations associated with distant lags, or (b) the contemporaneous effects are somehow legitimate, in which case it is impossible to figure out the extent to which x causes y versus the opposite (see also Geweke, 1982). Alternatively, if we suppose that unidirectional instantaneous causal effects are somehow ontologically real, then we should be able to estimate them by specifying lagged and contemporaneous effects in the same model (as you note), and with enough periods of observation combined with a small enough lag order, we can identify simultaneous contemporaneous causal effects of x and y on each other while at the same time allowing for contemporaneous covariance among x and y variables. Such a specification including AR effects (xt–>yt; xt-1–>yt; xt-1–>xt; yt–>xt; yt-1–>xt; yt-1–>yt; xtyt) should allow recovering lagged and contemporaneous effects by, effectively, using past observations as instruments in order to identify the simultaneity and the covariance. However, and interestingly, it seems that such a specification actually doesn’t help us get away from Granger’s position because by using the past lags as predictors the contemporaneous simultaneous effects can be understood as expressing the effect of past lags that are not accounted for elsewhere in the model. Whatever the case, perhaps there’s a simpler way to understand what Vaiseley and Miles are finding (and what many others find in relation to lags), which is possible when we move to a continuous-time interpretation of panel data models (see various excellent papers by Voelkle and Oud). From a CT perspective, we should be unsurprised by finding changing effects with different lag orders and even effects that change sign. This predictably occurs as a system oscillates over time. In turn, CT models can be used to uncover the continuous derivative functions that are being approximated in discrete-time panel data models. From a CT perspective, ‘biased’ estimates in many panel data models are actually correct in the context of predictable changes in effects over time as a system evolves, it’s just the case that a lagged effect at one time will in general not accurately estimate the lagged effect at another time. Just a few thoughts here. Curious to hear yours. Hope all is well, Mike
Interesting points. I’m not familiar with the CT models and their implications for discrete-time panel models.