4.4.1Introduction to Model Ensembles

What is an ensemble?

A single model has one view of the data and one pattern of mistakes. An ensemble combines the predictions of several models to produce one final prediction.

multiple models
      ↓
combine their predictions
      ↓
one final prediction

The main intuition is that different models often make mistakes on different samples. If their errors are not identical, combining their predictions can make some errors cancel out. Suppose three classifiers are each 80% accurate. They do not necessarily misclassify the same 20% of samples, so a majority vote can be more accurate than any one of them. This solves the exact problem that decision classifiers face when used alone.

Accuracy and diversity both matter

Useful ensemble members should ideally be:

• reasonably accurate on their own; and
• different enough to make different mistakes.

This second property is called diversity. Combining five nearly identical models often helps much less than combining five models whose errors differ.

You can think of this like averaging several noisy estimates: some estimates are too high, others are too low, and their random errors partly cancel. Averaging cannot, however, repair a systematic mistake shared by every model.

How predictions are combined

Regression: average the numbers

If $M$ regression models make predictions $\hat{y}_1,\ldots,\hat{y}_M$, their mean prediction is

$$\hat{y}=\frac{1}{M}\sum_{m=1}^{M}\hat{y}_m.$$

For example, predictions of 18, 21 and 24 give an ensemble prediction of 21.

Classification: vote or average probabilities

Hard voting asks every classifier for a class and chooses the class with the most votes.

Soft voting averages the predicted probabilities and chooses the class with the largest average probability. Suppose three models predict probabilities for classes 0 and 1:

The soft-voting ensemble predicts class 0. Unlike hard voting, it uses how confident each model is, not only its final class.

Weighted ensembles

Models do not have to contribute equally. A weighted ensemble uses

$$\hat{y}=\sum_{m=1}^{M}w_m\hat{y}_m, \qquad \sum_{m=1}^{M}w_m=1.$$

A stronger model can receive a larger weight. Choose models and weights using validation data—never by repeatedly checking the test set.

The main ensemble families

The most important distinction for the next chapters is between bagging and boosting.

Bagging: train independently, then average

Bagging is short for bootstrap aggregating. Its two ingredients are:

1. Bootstrap: create several training datasets by sampling rows with replacement.
2. Aggregate: train one model on each dataset, then vote or average their predictions.

If the original training set is

A  B  C  D  E

two bootstrap samples might be

A  C  C  E  B
D  A  D  B  E

Each sample usually contains as many rows as the original dataset. Because sampling is with replacement, a row can appear several times while another row may not appear at all.

                   ┌─ bootstrap sample 1 → model 1 ─┐
training data ─────┼─ bootstrap sample 2 → model 2 ─┼→ vote or average
                   └─ bootstrap sample 3 → model 3 ─┘

The models are independent once their datasets have been created, so they can be trained separately or in parallel.

Why bagging helps trees

A deep decision tree can be unstable: a small change in its training data can produce different splits and predictions. Bootstrap samples make each tree change in a different way. Averaging many trees allows some of that instability to cancel.

Bagging therefore mainly reduces variance. It does not magically fix a base model that is consistently wrong.

Different bootstrap samples create diversity. Random Forests add another source of randomness by letting each split consider only some of the features. The next module develops that idea fully.

Bagging versus ordinary averaging

An ordinary voting ensemble may combine different model types trained on the same data. Bagging usually trains copies of the same model type on different bootstrap samples. It is a particular way of creating diverse models before averaging them.

OptionalOptional: bootstrap samples and out-of-bag rows1 min readShowHide

For a training set with $n$ rows, the chance that one particular row is missed by all $n$ bootstrap draws is

$$\left(1-\frac{1}{n}\right)^n\approx e^{-1}\approx0.368.$$

For a reasonably large dataset, one bootstrap sample therefore contains about 63.2% of the original rows at least once and leaves about 36.8% out.

The unused rows are called out-of-bag samples for that model. Across a bagged ensemble, each training row is left out for some members. Their predictions can be combined into an out-of-bag estimate of model performance. The Random Forest chapter explains this properly.

Hide this optional section

Boosting: add corrections in sequence

Boosting takes a different approach. Its models are trained one after another, and each new model tries to improve what the current ensemble still gets wrong.

model 1 learns the broad pattern
        ↓
model 2 corrects some remaining errors
        ↓
model 3 adds another correction
        ↓
sum their contributions

Boosting often uses deliberately small decision trees, sometimes even one-split decision stumps. Each tree may be weak on its own, but the trees cooperate by correcting the current ensemble.

Two broad boosting ideas

• AdaBoost style: misclassified samples become more important, so the next learner pays more attention to them.
• Gradient boosting style: calculate the errors or residual-like corrections left by the current ensemble, then fit the next tree to improve them.

XGBoost, LightGBM and CatBoost are advanced systems for gradient-boosted trees. Their mathematics and implementation details belong in their dedicated chapters.

The additive view

A boosted model can be pictured as a starting prediction followed by a sequence of corrections:

$$F_M(x)=F_0(x)+\eta f_1(x)+\eta f_2(x)+\cdots+\eta f_M(x).$$

Here, $f_m$ is the new tree and $\eta$ is the learning rate, which controls the size of each correction.

• A large learning rate makes larger corrections and may need fewer trees, but can overfit or overshoot more easily.
• A small learning rate makes gentler corrections and usually needs more trees, but is often more robust.

The learning rate and number of trees must therefore be considered together.

Bagging versus boosting

A useful, but simplified, bias–variance picture is:

high variance
→ model reacts too much to its particular training sample
→ bagging averages many versions to reduce instability

high bias
→ model is too simple to capture the pattern
→ boosting adds simple learners to build a more flexible predictor

This is intuition rather than an absolute rule. The result still depends on the data, base models and settings.

Why trees work especially well in ensembles

Decision trees are common ensemble members because:

• small data changes can easily create different trees;
• trees capture non-linear relationships and feature interactions;
• deep trees have high variance, so averaging can help them greatly; and
• shallow trees make useful corrective learners for boosting.

This leads naturally to two major approaches:

unstable trees + averaging  → Random Forest
small corrective trees      → gradient boosting

Stacking: learn how to combine models

Stacking trains several base models, then uses their predictions as features for a final meta-model.

logistic regression prediction ─┐
random forest prediction ───────┼→ meta-model → final prediction
XGBoost prediction ─────────────┘

The training predictions supplied to the meta-model must be out-of-fold predictions. If a base model predicts the same rows it was trained on, its overly optimistic predictions leak training information into the meta-model.

OptionalOptional: blending1 min readShowHide

Blending is a simpler relative of stacking:

1. Reserve a validation set.
2. Train the base models on the remaining training data.
3. Predict the reserved set.
4. Train the meta-model on those predictions.

It is easier to organise, but less data-efficient than out-of-fold stacking because the meta-model sees only the reserved subset.

Hide this optional section

Ensembling across folds, seeds and settings

An ensemble does not have to combine completely different algorithms. In competitions, a simple and useful approach is to average predictions from the same model trained with different:

• cross-validation folds;
• random seeds; or
• hyperparameter settings.

For example, five fold-specific models can each predict the test data, after which their probabilities are averaged. Three XGBoost models with different seeds can be combined in the same way. These variations often make slightly different errors and improve robustness.

When ensembling does not help

An ensemble may add little when:

• all members make almost identical predictions;
• every model misses the same underlying pattern;
• some members are so poor that they weaken the result; or
• ensemble weights have been over-tuned to one validation split.

Ensembles also cost more computation and memory, make inference slower, and are usually harder to interpret and maintain than one model.

OptionalOptional mathematics: why error correlation matters1 min readShowHide

Suppose $M$ models have errors with the same variance $\sigma^2$ and every pair has error correlation $\rho$. The variance of their average is approximately

$$\operatorname{Var}(\text{average error}) =\sigma^2\left(\rho+\frac{1-\rho}{M}\right).$$

If the errors are independent, $\rho=0$ and the variance becomes $\sigma^2/M$. Averaging more members strongly reduces the noise.

If their errors are almost identical, $\rho\approx1$ and the variance remains close to $\sigma^2$ however many models are added. This is why both model quality and diversity matter.

Hide this optional section

The whole idea in one view

ensemble
multiple models → combine predictions → stronger or more stable result

voting / averaging
separate models → directly combine outputs

bagging
same model type → different bootstrap datasets
→ train independently → vote or average
main goal: reduce variance

boosting
simple learner → inspect remaining error → add correction → repeat
main goal: build a strong predictor sequentially

stacking
different models → predictions become features → meta-model

The next modules turn this map into concrete tree ensembles:

• Random Forests use bagging and extra randomness.
• XGBoost builds trees as sequential corrections.