Panel Data Explained: Fixed Effects, Random Effects and the Hausman Test

Understanding panel data — simply put

Panel data follows the same units, such as people, firms, regions or countries, over several periods. Because each unit is observed more than once, a regression can compare each unit with itself at different times instead of comparing different units with each other. That comparison removes every unit-specific factor that does not change over time, whether or not it can be measured. This is why panel data is a workhorse of applied microeconomics and of cross-country macroeconomics.

How does it actually help?

Take fuel prices and fuel use. Across Europe, countries with expensive fuel use less of it per person, and a simple cross-country OLS regression reads that as a strong price effect. The Netherlands differs from Spain in more than its pump price, though. It is denser and flatter, has a dense rail network and has far more cycling. Those features cut fuel use, and they also go together with high fuel taxes. Panel data lets the regression ask a sharper question: when the price of fuel rises in the Netherlands, how much does Dutch fuel use fall? For this example the model is ln q_it = α_i + β ln p_it + ε_it, where q_it is road fuel use per person in country i in year t, p_it the real pump price, α_i the fixed country effect and β the price elasticity: the percentage change in fuel use for a 1% change in price.

Why does this matter?

The elasticity drives policy. A finance ministry setting fuel duty, a climate agency projecting emissions under a carbon price and a central bank tracing how an oil shock feeds into spending all need to know how strongly demand responds to price. Reading the cross-country gap as the within-country response can overstate that number several times over. A government relying on it would overestimate how much a fuel tax cuts emissions and underestimate how much revenue it raises.

What makes fixed effects reliable?

Fixed effects remove only what is constant within a unit. They work when the remaining variation in x, the movement over time inside each country, is unrelated to the shocks that also move y in that country. Factors that change over time and hit every country at once, such as a pandemic or a continent-wide energy crisis, are handled by adding time effects. Factors that change over time inside one country and move with price are handled by neither, and need a different design.

The takeaway

Panel data turns “countries with high prices use less fuel” into “when prices rise, use falls by this much”, by comparing each country with itself. The rest of this guide works through a full example, then sets out the assumptions and the mechanics.

A worked example: fuel prices in six European countries

The table shows real pump prices and road fuel use per person for six European countries in 2019 and in 2022, the year energy prices spiked. The full sample covers every year from 2019 to 2024, giving 36 country-year observations. It is a teaching sample built to resemble the pattern in European pump-price and fuel-consumption data, and every statistic below is computed exactly from it.

Two patterns stand out. Across countries, Spain has the cheapest fuel and by far the highest use, while the Netherlands has the dearest fuel and the lowest use. Within every country, the 2022 price spike came with a fall in use. The question is which of the two comparisons measures the price elasticity.

Pooled OLS treats all 36 observations as if they came from one country and reports an elasticity of −0.98: a 10% price rise cuts fuel use by almost 10%. Most of that comes from the gap between countries. A regression of country averages on each other, a pure between-country comparison, gives −1.60. Adding country fixed effects compares each country only with itself and cuts the elasticity to −0.41.

Year effects go one step further. The 2022 energy crisis raised prices in all six countries at once, while a steady shift toward more efficient cars cut fuel use everywhere. Without year effects, the regression credits part of that common trend to price. The two-way estimate is −0.345, with a clustered standard error of 0.095 and a t-statistic of −3.65. A 10% rise in the price of fuel cuts use by about 3.5% within the year. The sample was built with a true elasticity of −0.30, and only the two-way fixed-effects estimate lies within one standard error of it.

Goodness of fit needs care in panels. The within R² of the two-way model is about 0.67: price and year effects explain two thirds of the variation in fuel use inside each country over time. The pooled R² of 0.69 looks similar but measures something different, mostly how well price sorts countries from high use to low.

Fixed or random effects?

Random effects treat each country's α_i as a random draw unrelated to price. When that holds, random effects use both within and between variation and give a more precise estimate. When it fails, they inherit part of the pooled bias. That happens here: the random-effects estimate of −0.68 sits between pooled OLS and fixed effects. The standard check is the Hausman test, which asks whether the fixed- and random-effects estimates differ by more than sampling error. A more robust version, due to Yair Mundlak, adds each country's average price to the random-effects regression. If country effects are unrelated to price, that average should not matter. Here its coefficient is −1.26 with a t-statistic of −7.8, so the random-effects assumption is rejected and fixed effects are the right choice. In small samples the classic Hausman statistic can come out negative, because the difference between two estimated variances need not be positive; it does in this sample, and the Mundlak form avoids the problem.

The assumptions

• Strict exogeneity. The shocks ε_it are unrelated to x in every period, past, present and future, not only the current one. This matters because the within transformation subtracts each country's average, and that average contains every period's x and ε. If a shock this year changes next year's price, for example when a demand surge leads a government to cut fuel duty, the demeaned price becomes correlated with the demeaned error and β̂ is biased.
• Variation within units. x must change over time inside each unit. Fixed effects use only within-unit changes, so a variable that never changes, such as geography or legal origin, is absorbed into α_i and its effect cannot be estimated. A variable that barely changes gives a very noisy estimate.
• Time-invariant heterogeneity. The omitted differences between units must be constant over the sample. α_i is a single number per unit. A factor that changes over time and moves with price, such as a rail expansion in one country, stays in ε_it and biases β̂ in the same way omitted variable bias distorts OLS.
• Correlated errors within units. Shocks to the same country in different years are usually correlated. Conventional standard errors assume every observation adds independent information. With serial correlation they overstate the information in the data, so standard errors should be clustered by unit. With only six clusters, as here, even clustered errors are imprecise, and applied work prefers dozens of units.

The formula

This section sets out the estimators formally. Readers who need only the intuition can skip to the limitations. The general model with K regressors is:

The within (fixed-effects) estimator. Set λ_t = 0. Averaging the model over time for each unit gives ȳ_i = α_i + x̄_i′β + ε̄_i. Subtracting this from the original equation removes α_i:

OLS on the demeaned data gives the fixed-effects estimator. It is numerically identical to adding a dummy variable for every unit, the least squares dummy variable (LSDV) regression:

Two-way fixed effects. Adding λ_t, in a balanced panel, subtracts the unit mean and the period mean and adds back the grand mean:

Random effects. If α_i is uncorrelated with x_it and has variance σ²_α, the composite error α_i + ε_it is correlated within units. Feasible generalized least squares removes only a fraction θ of each unit mean:

The Hausman test. Under the null hypothesis that α_i is uncorrelated with the regressors, both estimators are consistent and random effects is efficient. Under the alternative, only fixed effects is consistent:

Mundlak's version adds the unit means to the random-effects model, y_it = μ + x_it′β + x̄_i′γ + α_i + ε_it, and tests γ = 0 with a cluster-robust Wald test. The estimate of β from this regression equals the fixed-effects estimate. In the example both are −0.345.

When panel data isn't the right tool

• Time-varying confounders. Fixed effects do nothing about omitted factors that change over time within units. If the regressor is driven by such factors, an instrumental variables design is needed.
• Policy changes at a known date. When some units are treated by a policy and others are not, difference-in-differences, a two-way fixed-effects model with a treatment indicator, makes the parallel-trends assumption explicit. When treatment starts at different dates in different units, the standard two-way estimator can be biased, and newer staggered-adoption estimators are preferred.
• Dynamics. If last year's outcome affects this year's, as when fuel use depends on the existing car stock, adding y_i,t−1 to a fixed-effects model creates a bias of order 1/T, known as Nickell bias. Short panels then need generalized method of moments (GMM) estimators such as Arellano–Bond.
• Long, trending panels. With many periods and trending variables, panel regressions face the same spurious-regression risk as single series, and the unit-root and cointegration methods of time series analysis apply.

The takeaway

Fixed effects buy credibility by throwing away the between-unit comparison and keeping only changes within units. Add time effects for shocks common to all units, cluster standard errors by unit, and test random effects before relying on them. What fixed effects cannot fix is a confounder that changes over time.

Further reading

• Understanding OLS Regression — internal
• Difference-in-Differences Explained — internal
• Instrumental Variables Explained — internal
• Time Series Analysis Explained — internal
• European Commission: Weekly Oil Bulletin (pump prices) — external reference
• Eurostat: energy statistics (fuel consumption) — external reference

Data note: the worked example is a teaching sample constructed by EconoLens to resemble the pattern in European pump-price data from the European Commission's Weekly Oil Bulletin and fuel-consumption data from Eurostat. The figures are illustrative, not the official series; a production version would pull the live series. All estimates were computed by EconoLens.