Time Series Modeling: Why ARIMA Models Beat Linear Regression for Temporal Data

Accurate forecasting begins with understanding temporal dependence. Build on the concepts in this blog to examine key assumptions and start fitting your own models by joining Fundamentals of Time Series Analysis with Professor Daniel J. Henderson, November 12-14.

Introduction

When analyzing data that changes over time, many analysts instinctively reach for linear regression. It’s familiar, intuitively interpretable, and widely understood. However, linear regression fundamentally misunderstands the nature of time series data. In this post, we’ll explore why ARIMA (AutoRegressive Integrated Moving Average) models are specifically designed for temporal data and demonstrate their superiority when it comes to forecasting temporal data.

Linear regression makes a critical assumption: observations are independent of each other. But time series data violates this assumption in a fundamental way: today’s value is often related to yesterday’s value, last week’s value, or seasonal patterns.

Consider this simple example: if you’re trying to predict tomorrow’s stock price, would you rather know:

• Linear regression approach: Various economic indicators, company metrics, etc.
• Time series approach: What the stock price was today, yesterday, and its recent trend.

Both approaches have merit, but they’re solving fundamentally different problems. Linear regression asks “What variables explain this outcome?” while ARIMA asks “How does this variable’s past predict its future?”

Since the goal of this post is to compare these approaches when it comes to predictions, we will fit our models to data from 1949-1959 and then predict the values for 1960. Comparing these predicted values to the actual observed values will give us a good idea of the model’s performance in a prediction scenario.

We will explore these assumptions and how they manifest themselves by comparing the results of both approaches. To this end, we use the classics AirPassengers data set, which shows monthly airline passenger numbers from 1949-1960.

LEARN MORE IN A SEMINAR

Setting Up the Data

# Load required libraries
library(forecast)
library(tidyverse)
library(tseries)

data("AirPassengers")

air_df <- tibble(
  time = as.numeric(time(AirPassengers)),
  passengers = as.numeric(AirPassengers),
  month = cycle(AirPassengers)
) |> 
  mutate(type = case_when(
    time >= 1960 ~ "Test",
    TRUE ~ "Training"
  )) |> 
    rowid_to_column(var = "time_trend")

glimpse(air_df)

Rows: 144
Columns: 5
$ time_trend <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, …
$ time       <dbl> 1949.000, 1949.083, 1949.167, 1949.250, 1949.333, 1949.417,…
$ passengers <dbl> 112, 118, 132, 129, 121, 135, 148, 148, 136, 119, 104, 118,…
$ month      <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7,…
$ type       <chr> "Training", "Training", "Training", "Training", "Training",…

Figure 1: Air passengers time series data (1949-1960)

The data clearly shows an upward trend with seasonal patterns that intensify over time.

Approach 1: Linear Regression (The Wrong Way)

# Naive linear regression approach
lm_model <- lm(passengers ~ time_trend + factor(month), 
               data = air_df |> filter(type == "Training"))
               
# Generate predictions
air_df$lm_pred <- predict(lm_model, air_df)

Figure 2: Linear regression fit vs. actual data

We can see that for earlier years, the model overpredicts values during peak months and underpredicts during slower months. After the half-point, around 1955, the model starts to underpredict peak months while showing comparably good fit for the slower months.

The Problem: Autocorrelated Residuals

The shifting pattern we observe in the linear regression results – where the model overpredicts peak months in early years but underpredicts them later – is a symptom of autocorrelated residuals. When residuals (prediction errors) are autocorrelated, it means the errors follow predictable patterns over time rather than being random, indicating that our model is systematically missing temporal information that could improve predictions. This is particularly problematic for forecasting because these systematic errors compound over time, making your 1960 predictions highly unreliable.

We can test for residual autocorrelation using the Box-Ljung test:

ljung_box_lm <- Box.test(residuals(lm_model), lag = 12, type = "Ljung-Box")

The p-value for the Ljung-Box test is reported as 0, which means it is some very small number, much less than the usual .05 criterion for statistical significance. This indicates significant autocorrelation in residuals, violating a key assumption of linear regression.

Approach 2: ARIMA Model (The Right Way)

ARIMA (AutoRegressive Integrated Moving Averages) models offer a good way for us to obtain reliable estimates.

Before we dive into comparisons, let’s break down what ARIMA actually means and why each component matters:

ARIMA(p,d,q) stands for:

• AR(p) – AutoRegressive: Uses p previous values to predict the current value.
  • Think: “Today’s weather is similar to yesterday’s weather.”
  • Formula: Y t = ϕ 1 Y t − 1 + ϕ 2 Y t − 2 + … + ϕ p Y t − p + ε t
• I(d) – Integrated: Differences the data d times to make it stationary.
  • Think: “Remove trends by looking at changes rather than levels.”
  • First difference: Y t − Y t − 1 removes linear trends.
• MA(q) – Moving Average: Uses q previous error terms to predict current value.
  • Think: “If we were wrong yesterday, adjust today’s prediction accordingly.”
  • Formula: Y t = ε t + θ 1 ε t − 1 + θ 2 ε t − 2 + … + θ q ε t − q

For seasonal data, like here, we add ARIMA(P,D,Q)ₛ components:

• SAR(P) – Seasonal AutoRegressive: Uses P previous seasons to predict current value.
  • Think: “This July’s sales are similar to last July’s sales.”
  • Formula: Y t = Φ 1 Y t − s + Φ 2 Y t − 2 s + … + Φ P Y t − P s + ε t
• SI(D) – Seasonal Integrated: Differences data at seasonal lags D times.
  • Think: “Remove seasonal trends by comparing to same month last year.”
  • Seasonal difference: Y t − Y t − 12
• SMA(Q) – Seasonal Moving Average: Uses Q previous seasonal error terms.
  • Think: “If we were wrong last summer, adjust this summer’s prediction.”
  • Formula: Y t = ε t + Θ 1 ε t − s + Θ 2 ε t − 2 s + … + Θ Q ε t − Q s

Why Each Component Matters

• AR terms capture short-term momentum and persistence in the data.
• I terms handle trending data that isn’t stationary.
• MA terms capture short-term irregularities and shock effects.
• SAR terms capture seasonal momentum (this January like last January).
• SI terms handle growing seasonal patterns and seasonal trends.
• SMA terms capture seasonal shock effects and irregular seasonal variations.

Model Selection Intuition

Instead of playing around with these parameters, we can use the auto.arima() function in R. This function tests different combinations and picks the model that:

1. Makes residuals look like white noise (uncorrelated).
2. Finds the best yet most parsimonious model by comparing information criteria (AIC/BIC).
3. Passes diagnostic tests for model adequacy.

The only pre-condition is that we need to reshape our data into a ts object. Our data follow a 12-month cycle, which we have to specify.

ts_passenger <- air_df |> 
  filter(type == "Training") |> 
  pull(passengers) |> 
  ts(start = c(1949, 1), frequency = 12)
# Auto-fit ARIMA model
arima_auto <- auto.arima(ts_passenger, seasonal = TRUE)
summary(arima_auto)

Series: ts_passenger 
ARIMA(1,1,0)(0,1,0)[12] 

Coefficients:
          ar1
      -0.2431
s.e.   0.0894

sigma^2 = 109.8:  log likelihood = -447.95
AIC=899.9   AICc=900.01   BIC=905.46

Training set error measures:
                   ME     RMSE      MAE       MPE     MAPE      MASE       ACF1
Training set 0.579486 9.907267 7.483159 0.1187348 2.880429 0.2457523 0.01227544

Now we can easily extract the fitted values from our model and forecast data for the next 12 months (our held-out test set). The next plot compares the results from the linear model to the ARIMA model.

# Generate fitted values and forecasts
air_df$arima_fitted <- c(fitted(arima_auto), forecast(arima_auto, h = 12)[["mean"]])

Figure 3: Linear regression and ARIMA fit vs. actual data

Interpreting the ARIMA Model

The auto.arima() function selected an ARIMA(1,1,0)(0,1,0)[12] model for the log-transformed passenger data. This is actually a beautifully parsimonious model. Let’s delve into what this means:

• AR(1): Current month depends on the previous month with coefficient -0.2431.
• I(1): One level of differencing removes the growth trend over time.
• No MA terms: Recent prediction errors don’t influence future forecasts.
• Seasonal (0,1,0)[12]: Simple seasonal differencing captures yearly patterns without complex seasonal terms.

The negative AR coefficient (-0.2431) is particularly interesting. It suggests a mean-reverting pattern after differencing. In business terms: if passenger growth was unusually high last month (after accounting for trend and seasonality), this month will likely see a modest pullback toward the long-term pattern.

Model performance: With a Mean Average Prediction Error (MAPE) of 2.88%, predictions are typically within 3% of actual values. This is excellent accuracy for forecasting. The clean residuals (ACF1 ≈ 0) confirm the model has captured the essential temporal patterns.

Conclusion

Time series data tells a story that unfolds through time: each observation carries information from the past that helps predict the future. Linear regression, with its assumption of independence, however, is more like trying to understand a conversation by reading individual sentences in random order. ARIMA models, in contrast, understand that the narrative flow matters.

Our analysis reveals this fundamental difference in stark terms. Where linear regression sees seasonal noise around a trend, ARIMA recognizes the evolving patterns of human behavior. The growing prosperity of the 1950s paired with more affordable air fares didn’t just increase travel, it also amplified seasonal preferences. The model’s simplicity and higher accuracy demonstrates that respecting temporal structure often leads to both better predictions and deeper insights.

What does this mean for you? When your data has a temporal dimension, honor it. Check for autocorrelation, test your residuals, and let the data’s natural timeline guide your modeling choices. Your forecasts – and, hence, your decisions – will be far more reliable when you choose tools designed for the structure your data actually has.

This simple ARIMA model has significantly outperformed linear regression. For more complex tasks, the ARIMA model can be extended. ARIMAX models, for instance, have the ability to incorporate exogeneous variables (such as, in this case, air fares and average household income). Join our upcoming seminar on Fundamentals of Time Series Analysis with instructor Professor Daniel J. Henderson to learn more about the underlying assumptions and how you can get started with fitting your own models.