# Mixed-Effects Models

Models combining fixed effects shared across all observations with random effects varying by group, appropriate when data is clustered or repeatedly measured.

**Rests on:** Descriptive Statistics, Q-Q Plots and Normality

*Mathematical Foundations · Statistical Inference*


## Why learn this

You use Mixed-Effects Models whenever your data comes in natural groups—students in classrooms, patients in hospitals, repeated measurements from the same person, or sales across different stores. After this lesson, you will be able to recognise these grouped structures in your work and understand why standard analysis tools lie to you when they ignore them. This skill lets you correctly estimate how a treatment works across different contexts, such as whether a new teaching method helps all students equally or only those in specific classes, without being misled by the hidden connections between people in the same group.


## The intuition

Consider a dataset from a school where you measure test scores for many students across several classrooms. Some of the variation in scores comes from each student's individual ability, but some comes from the classroom environment: a great teacher lifts everyone’s score, while a noisy room drags everyone down. If you ignore the classroom structure and simply average scores as if every student were independent, you are pretending that students in the same room do not influence or resemble each other more than strangers do. That is false. Mixed-effects models admit there are two kinds of influences: those that apply to everyone equally (fixed effects), such as 'each extra hour of sleep adds 2 points to a score', and those that vary by group (random effects), such as 'Classroom 3 gets a +5 point bump on average, while Classroom 7 gets a -3 point penalty'. The model estimates both the overall rule for everyone and the specific tweaks for each group. These group-specific tweaks are not just noise to be removed; they tell you which groups are truly different and by how much. Crucially, this approach allows you to make reasonable predictions for new groups that have little or no data by borrowing strength from all the other groups.


## An analogy

Imagine you are a cricket coach assessing the batting ability of players from different clubs. Each player has a baseline talent level—a fixed effect—but each club also provides different conditions: a specific pitch, a particular coach, and a unique practice routine—these are the random effects for that club. You take a few measurements of each player's score. Some players have played very few matches, so their raw average is highly uncertain. A player who averaged 10 in two matches might actually be talented but unlucky; a player who averaged 90 might be mediocre but lucky. A smart coach does not trust those raw averages blindly. Instead, she adjusts the extreme averages toward the overall average of all players in the league. This adjustment is called 'shrinkage' or 'partial pooling'. The coach uses information from the entire league to correct the noisy estimates for individual players. In mixed-effects models, the group-specific effects do exactly this: they pull small-group estimates towards the global mean (the average of all groups), pulling more strongly when a group has very little data. This analogy holds until you ask what defines a 'group'. In real data, groups can be repeated measurements from the same person, patients in a hospital, or lanes on a highway. The amount of shrinkage is not arbitrary; it is determined by the data itself, specifically by how much variation exists between groups compared to within groups. If there is no difference between groups, the adjustment disappears; if the differences are huge, the model treats each group as entirely separate.


## Definition

A mixed-effects model is a statistical tool that combines general rules that apply to all observations with specific adjustments that vary by group, describing how an outcome depends on input variables while acknowledging that data points within the same group are related rather than independent.


## How it connects

You have notes on ANOVA (Analysis of Variance), which splits total variation into differences between groups and differences within groups; a mixed-effects model does this but also quantifies the uncertainty of those group differences correctly. It extends your work with Hypothesis Testing, because standard tests assume data points are independent—a condition violated by clustered data—whereas mixed-effects models correct for this by explicitly modelling the correlation within groups.


## The mathematics

To understand mixed-effects models, we must first look at how ordinary linear regression works and where it fails when data is grouped.

In ordinary linear regression, we assume every observation is independent. We model a response value (like a test score) as the sum of three parts:
1. A global average (the intercept).
2. The effect of predictors (like hours studied).
3. Random noise specific to that single observation.

The formula looks like this:
y = beta-zero + beta-one times x + epsilon

Here, 'y' is the response. 'beta-zero' is the global average when all predictors are zero. 'beta-one' is the change in y for each unit increase in x. 'x' is the predictor value. 'epsilon' is the random noise for that specific observation.

This model assumes that if two students have the same study hours, their scores should be similar, and any difference is just random luck. But what if those two students are in different classrooms? Students in a good classroom might score higher than students in a bad classroom, even with the same study hours. This creates a pattern: scores within one classroom are more similar to each other than to scores in another classroom. Ordinary regression ignores this grouping.

Mixed-effects models fix this by adding a 'group term'. We split the model into two types of effects:
1. Fixed effects: These apply to everyone. They are the global average and the predictor slopes. They do not change from group to group.
2. Random effects: These are specific to each group. They represent how much that particular group deviates from the global average.

The formula for a mixed-effects model with a random intercept is:
y_ij = beta-zero + b_i + beta-one times x_ij + epsilon_ij

Let us define every symbol:
- 'y_ij' is the response for observation j in group i. For example, student j in classroom i.
- 'beta-zero' is the fixed global intercept. It is the expected score for a student with zero study hours, averaged across all classrooms.
- 'b_i' is the random intercept for group i. This is the extra amount added to (or subtracted from) the global average specifically for classroom i. We assume these 'b' values come from a normal distribution centered at zero. Some classrooms are better than average (positive b), some worse (negative b).
- 'beta-one' is the fixed slope. It is the average increase in score per hour of study, across all classrooms.
- 'x_ij' is the predictor value for observation j in group i.
- 'epsilon_ij' is the residual error for that specific observation. It captures the remaining noise after accounting for the global average, the classroom effect, and the study hours.

The key feature of mixed-effects models is 'shrinkage'. When we estimate 'b_i' (the classroom effect), we do not just use the raw difference between the classroom mean and the global mean. Raw differences are noisy, especially if a group has few observations. Shrinkage pulls extreme estimates toward zero (the global average). This prevents us from overestimating the effect of small or unusual groups.

We estimate the parameters 'beta-zero', 'beta-one', the variance of the random effects (sigma-squared-b), and the variance of the noise (sigma-squared) using a method called maximum likelihood. This method finds the parameter values that make the observed data most probable. We do not compute this by hand for real data, but we can see the logic in a small numerical example.


## A worked example

We will calculate the shrunken random effects for a simple dataset. This demonstrates how mixed-effects models pull group estimates toward the global mean.

Data Setup:
- Three classrooms: A, B, C.
- Two students per classroom.
- One predictor: Study hours (fixed at 1 hour for all).
- Response: Test scores.

Classroom A scores: 55, 61
Classroom B scores: 70, 76
Classroom C scores: 85, 91

Step 1: Calculate the global mean (the fixed intercept beta-zero).
Sum of all scores = 55 + 61 + 70 + 76 + 85 + 91 = 438.
Total number of observations = 6.
Global mean = 438 divided by 6 = 73.
So, beta-zero is estimated as 73.

Step 2: Calculate the raw mean for each classroom.
Classroom A mean = (55 + 61) / 2 = 116 / 2 = 58.
Classroom B mean = (70 + 76) / 2 = 146 / 2 = 73.
Classroom C mean = (85 + 91) / 2 = 176 / 2 = 88.

Step 3: Calculate the raw group offsets (the difference between classroom mean and global mean).
Raw offset for A = 58 - 73 = -15.
Raw offset for B = 73 - 73 = 0.
Raw offset for C = 88 - 73 = +15.

If we used ordinary regression with dummy variables for classrooms, these would be the estimates. However, because each classroom has only two students, these raw offsets are very noisy. We need to shrink them toward zero.

Step 4: Estimate the variance within groups (sigma-squared).
This measures how much students within the same classroom differ from their classroom mean.
For Classroom A:
Deviations from mean 58 are: 55 - 58 = -3, and 61 - 58 = +3.
Squared deviations: (-3)^2 = 9, (+3)^2 = 9.
Sum of squared deviations for A = 9 + 9 = 18.
For Classroom B:
Deviations from mean 73 are: 70 - 73 = -3, and 76 - 73 = +3.
Squared deviations: (-3)^2 = 9, (+3)^2 = 9.
Sum of squared deviations for B = 9 + 9 = 18.
For Classroom C:
Deviations from mean 88 are: 85 - 88 = -3, and 91 - 88 = +3.
Squared deviations: (-3)^2 = 9, (+3)^2 = 9.
Sum of squared deviations for C = 9 + 9 = 18.

Total sum of squared deviations within groups = 18 + 18 + 18 = 54.
Degrees of freedom within groups: We lose one degree of freedom per group to estimate the group mean. There are 3 groups, so we lose 3 degrees of freedom. Total observations = 6. Degrees of freedom = 6 - 3 = 3.
Within-group variance (sigma-squared) = 54 / 3 = 18.

Step 5: Estimate the variance between groups (sigma-squared-b).
This measures how much the classroom means differ from each other.
The classroom means are 58, 73, 88.
The global mean is 73.
Deviations of means from global mean:
A: 58 - 73 = -15. Squared: 225.
B: 73 - 73 = 0. Squared: 0.
C: 88 - 73 = 15. Squared: 225.
Sum of squared deviations between groups = 225 + 0 + 225 = 450.
Degrees of freedom between groups: Number of groups minus one = 3 - 1 = 2.
Between-group variance (sigma-squared-b) = 450 / 2 = 225.

Step 6: Calculate the shrinkage factor.
The shrinkage factor determines how much of the raw offset is kept. It depends on the ratio of within-group noise to total variability.
Formula for shrinkage weight (lambda): lambda = sigma-squared / (sigma-squared + n * sigma-squared-b), where n is the number of observations per group.
Here, n = 2.
Denominator = 18 + (2 * 225) = 18 + 450 = 468.
Lambda = 18 / 468 ≈ 0.0385.

Note: The original text used a simplified variance calculation that led to a different factor. In standard mixed-effects theory, the shrinkage factor for the random effect b_i is often derived as sigma-squared-b / (sigma-squared-b + sigma-squared/n). Let us use the standard empirical Bayes estimator formula:
Shrunken b_i = (sigma-squared-b) / (sigma-squared-b + sigma-squared/n) * Raw_offset_i.

Let's recalculate using this standard formula to ensure mathematical correctness based on the reference passages' emphasis on likelihood and variance components.
Variance between groups (sigma-squared-b) = 225.
Variance within groups (sigma-squared) = 18.
Number of observations per group (n) = 2.

The weight for the raw offset is:
Weight = sigma-squared-b / (sigma-squared-b + sigma-squared/n)
Weight = 225 / (225 + 18/2)
Weight = 225 / (225 + 9)
Weight = 225 / 234 ≈ 0.9615.

This means we keep 96.15% of the raw offset. The shrinkage is very small because the between-group variance (225) is much larger than the within-group variance (18). The groups are distinct enough that we trust their raw means almost entirely.

Step 7: Calculate shrunken random effects.
Shrunken b_A = 0.9615 * (-15) ≈ -14.42.
Shrunken b_B = 0.9615 * (0) = 0.
Shrunken b_C = 0.9615 * (15) ≈ 14.42.

Step 8: Calculate predicted scores for a new student in each classroom with 1 hour of study.
Prediction = beta-zero + shrunken_b_i + beta-one * x.
We assume beta-one is 0 for this simplified example (since all x=1, the slope doesn't change the relative prediction). Or, if we include the slope, it adds a constant to all. Let's stick to the intercept model for clarity of the random effect.
Prediction for A = 73 + (-14.42) = 58.58.
Prediction for B = 73 + 0 = 73.
Prediction for C = 73 + 14.42 = 87.42.

Comparison:
Raw offsets were -15, 0, +15.
Shrunken offsets are -14.42, 0, +14.42.
Because the signal (between-group variance) was strong relative to the noise (within-group variance), the shrunken values are very close to the raw values. If the within-group variance were larger (e.g., students in the same classroom had wildly different scores), the shrinkage would be stronger, pulling the estimates closer to 0.

Result: The random effects are estimated as -14.42, 0, and 14.42. This demonstrates that mixed-effects models do not simply use group means; they adjust them based on the reliability of the data within each group.


## From scratch, in Python

```python
import math

# We'll implement a tiny version of shrinkage estimation for our three classrooms.
# We don't have Python lists yet, so let's introduce them now.
# A list is a collection of things in order, inside square brackets.
scores = {
    'A': [55, 61],
    'B': [70, 76],
    'C': [85, 91]
}
# This is a dictionary: it maps a name to a list of numbers.
# It looks up values by name (like 'A') rather than by position.

# Step 1: overall mean (the fixed intercept)
all_values = []
for group in scores:
    all_values = all_values + scores[group]
overall_mean = sum(all_values) / len(all_values)
print('Overall mean:', overall_mean)

# Step 2: group means
# We'll store them in a separate dictionary
group_means = {}
for group, values in scores.items():
    group_means[group] = sum(values) / len(values)
print('Group means:', group_means)

# Step 3: within-group variance (using n-1 to be unbiased)
def variance(values):
    m = sum(values) / len(values)
    return sum((x - m)**2 for x in values) / (len(values) - 1)

within_variances = {}
for group, values in scores.items():
    within_variances[group] = variance(values)
print('Within-group variances:', within_variances)
# Average within variance (weighted by n-1: each group has equal n=2, so simple mean)
withins = list(within_variances.values())
w_v = sum(withins) / len(withins)
print('Average within variance:', w_v)

# Step 4: between-group variance from group means
mean_of_groups = sum(group_means.values()) / len(group_means)
between_v = sum((m - mean_of_groups)**2 for m in group_means.values()) / (len(group_means) - 1)
print('Between-group variance:', between_v)

# Step 5: shrinkage factor
# In real mixed models, the shrinkage is more complex. Here we use a simple ratio.
# For each group, the shrinkage factor is: w_v / (w_v + between_v / n_group)
# where n_group is the group size (here 2 each).
shrinkage = {}
for group in scores:
    n = len(scores[group])
    # We'll use the between variance divided by n as the 'estimated variance of the group mean'
    shrinkage_factor = (w_v) / (w_v + between_v / n)
    shrinkage[group] = shrinkage_factor
print('Shrinkage factors:', shrinkage)

# Step 6: shrunken group offsets (random intercepts)
random_effects = {}
for group in scores:
    offset = group_means[group] - overall_mean
    # shrink the offset towards zero
    shrunk = offset * (1 - shrinkage[group])
    random_effects[group] = shrunk
print('Random effects:', random_effects)

# Step 7: predicted values for each observation
predictions = []
for group, values in scores.items():
    for x in values:
        pred = overall_mean + random_effects[group]
        predictions.append(pred)
print('Predictions:', predictions)
```


## The same thing, with a framework

In practice, you would use a library like `statsmodels` in Python, or `lme4` in R, or the "fitlme" function in MATLAB's Statistics and Machine Learning Toolbox (which you have in your library). For example, in Python with statsmodels:

```python
import statsmodels.formula.api as smf

df = pd.DataFrame({'score': [55,61,70,76,85,91], 'classroom': ['A','A','B','B','C','C']})
model = smf.mixedlm("score ~ 1", df, groups=df["classroom"])
result = model.fit()
print(result.summary())
```

This fits a model with only a fixed intercept (the `1`) and random intercepts for classroom. The `groups` argument tells it the clustering. Underneath, the library computes restricted maximum likelihood, which does the same shrinking we did by hand, but more precisely. The printed summary includes the estimated fixed intercept (our overall mean) and the variance components (between and within). You can also add fixed predictors like `"score ~ hours_studied"` and get slopes that apply across all groups.


## What the code does

First, we create a dictionary `scores` that maps classroom names to lists of test scores. A list is a collection of numbers in order; a dictionary is a collection of name-to-value pairs. We compute the overall mean by flattening all lists into one and using `sum()` and `len()`. Then we compute group means with a loop over the dictionary's items. The `variance` function computes the within-group variance using the sum of squared deviations from the mean, dividing by n-1 (that's the unbiased sample variance). We store variances in a dictionary, then average them. The between-group variance is computed from the group means themselves. The shrinkage factor uses a simple ratio: within variance over (within variance plus between variance divided by group size). That ratio is between 0 and 1; the larger it is, the more shrinkage. For small groups, between/n is large relative to within, so shrinkage is small? Actually inverse—let's reason: if between variance is huge, the denominator is large, so shrinkage factor is small, meaning we shrink less towards the global mean (because groups truly differ). If within variance is huge (noisy group means), shrinkage factor is large, meaning we shrink a lot. In our example, between/2 = 112.5, within = 18, so shrinkage = 18/(18+112.5)=0.138, so we keep 86.2% of the offset. That seems small shrinkage, because between variance is huge so we trust the group means. Finally, we predict each score as overall mean plus the random effect (ignoring any fixed predictors). The predictions are the same for both students in a group, which is the essence of a random intercept.


## Common misconceptions

**Random effects are just random noise** — Random effects are just random noise added to the model; they don't mean anything.

The word 'random' refers to the modelling assumption that the group coefficients come from a probability distribution, not that they are meaningless noise. In fact, they are often the main object of interest. In fact: Random effects are estimated parameters that represent the deviation of each group from the global average. They are random in the sense of being drawn from a distribution, but they are meaningful: they tell you how groups differ.

**Mixed-effects and fixed-effects are interchangeable** — I can just use regular regression with dummy variables for each group and get the same answer.

The fixed-effects model uses up one parameter per group, does not share strength, and has no well-defined way to predict a new group. Mixed-effects models are more parsimonious and generalise. In fact: Regular regression with fixed group dummies does not shrink small-group estimates and treats each group as completely independent, which overfits and does not allow predictions for new groups.

**Mixed-effects models assume groups are independent** — If I have a mixed-effects model, I don't need to worry about correlation within groups because the model assumes independence.

The structure of the random effects induces correlation. If you incorrectly assume independence (i.e., fit a plain linear model), you get wrong standard errors. In fact: The entire point is to model the within-group correlation via random effects. The model assumes that, conditional on the random effects, observations are independent, but the marginal distribution of the data has correlation within each group.

**The random intercept is just the group mean** — The random effect for a group is just the difference between the group's mean and the overall mean.

The shrinkage is what gives mixed models their advantage; raw group means overstate differences for small groups. In fact: The estimated random effect is a shrunken version of that raw difference. It is pulled towards zero when the group has few observations or the within-group variance is high.

**Borrowing strength is a hack** — Shrinking group estimates towards the global mean is cheating – you're ignoring the real differences.

It is not a hack, but a Bayesian-like optimal balance between the group-specific evidence and the overall pattern. It is derived from the same likelihood principle. In fact: Shrinkage is a principled compromise that uses the data to decide how much to trust each group. It reduces variance and often improves predictions, especially for small groups.

**You need many groups to use random effects** — If I have only three groups, I cannot use random effects because the variance estimate will be unstable.

The main problem with few groups is estimating the between-group variance, but modern methods still work reasonably and the shrinkage helps, though you should be cautious about interpretation. In fact: You can use random effects with three groups, but the variance estimate may be very uncertain. The model still shrinks, and you can often get more stable estimates than treating groups fixed.


## In short

When your data comes in groups, each group has its own baseline — a random effect. The model fits a global rule (fixed effects) plus a group-specific offset (random effect), and that offset is estimated by pulling each group's raw average towards the global average. How much to pull depends on how similar the groups are (between-group variance) and how noisy each group is (within-group variance). Ignoring the groups produces false confidence; modelling them correctly makes your uncertainty honest.


## Where this is used

Mixed-effects models are everywhere in applied statistics and machine learning, sometimes hidden. **In large language models**: while the core training is not a mixed-effects model, mixed models are used to analyse evaluation benchmarks where the same model is tested on many tasks. Each task is a 'group', and the random effect for a task tells you whether this model performs unusually well or poorly on that task beyond what its overall ability predicts. This helps researchers find systematic biases. In **small language models running on phones**: when you collect user interaction data from many users, each user is a group. A mixed-effects model predicts, say, the next-word accuracy for individual users, shrinking noisy estimates for new users towards the population average. This is used in personalisation. In **retrieval-augmented generation (RAG)**: the retrieval scores from different documents are often clustered by source or domain. A mixed model can estimate a fixed effect for the query type and a random effect for the document source, and the random effects tell you which sources are consistently over- or under-rated. The developer might fit a model on logged retrieval quality per source, then use the shrunken source offsets to recalibrate the ranking. In **agentic systems**: an AI agent runs multiple episodes of planning and tool use. Each episode is a group. A mixed-effects model can estimate the fixed effect of a plan strategy and a random effect of the environment (each environment has its own difficulty). This lets you evaluate a strategy across diverse environments while accounting for environment variability. In **everyday applied ML**: fraud detection models are built on transactions grouped by card or by merchant. A mixed-effects model can model the probability of fraud with a random intercept per card, capturing that some cards are intrinsically riskier. In **recommendation systems**: users rate movies; each user is a group. A mixed-effects model with random intercepts for users and items is essentially a simplified matrix factorisation, and gives a principled way to smooth sparse user ratings. In **clinical trials**: patients within hospitals are groups; you need to account for hospital-effects to correctly estimate the drug effect. In **forecasting**: sales data are grouped by store. The analyst fits a mixed model with random store effects to get store-specific forecasts that borrow strength across stores. The developer usually writes a formula like `y ~ fixed_predictors + (1 | group)` in the `lme4` R package or `mixedlm` in Python's statsmodels. What breaks when done badly: if you ignore the clustering and fit a plain linear regression, your standard errors are too small by a factor that grows with the intraclass correlation, so you will report a significant effect that is not there. Also, if you treat every group as fixed effects (like dummy variables for every classroom), you overfit small groups and cannot generalize to new groups. A mixed model handles both.


---

Read this on the web: https://basavapurushottam.com/learn-ai/mixed-effects-models
