4.5.3Loss Functions

When a model trains, it needs a single number that says how wrong its current predictions are, so that it can adjust its parameters to make that number smaller. That number is the loss. Every model you have met so far has one, even where it was not named: linear regression minimises the squared error, and logistic regression minimises the log loss.

Choosing a loss is a real modelling decision, not a technical detail. The same model trained with two different losses can learn quite different things from the same data: one version aims at the average label and another at the median label, and a single extreme label can pull the first far more than the second.

Most of the time a model's default loss is a sensible choice. This module helps you recognise when it is not, for example when the labels contain outliers, when some mistakes cost more than others, or when the task's metric rewards something that the default loss does not.

A model is trained by choosing the parameters that minimise a loss function on the training data, as introduced in Terminology, and it is judged by a metric, which often differs from the loss. This module compares the common losses, relates them to assumptions about noise, shows how weights enter a loss, and gives their names in scikit-learn and XGBoost.

Loss, cost and objective

Here $f(x_i; \theta)$ is the model's prediction for sample $i$, $R(\theta)$ is a penalty such as the sum of squared weights, and $\lambda \ge 0$ sets its strength. The $\alpha$ of Ridge and Lasso and the $C$ of logistic regression are such strengths. Many texts say "loss" for all three terms when the meaning is clear from context.

Three samples with labels 3, 5 and 10 and predictions 4, 5 and 7 give the following losses.

The squared-error cost is $(1 + 0 + 9)/3 \approx 3.33$, the mean squared error. The absolute-error cost is $(1 + 0 + 3)/3 \approx 1.33$, the mean absolute error. Sample 3 accounts for 90% of the first cost and 75% of the second: squaring gives large errors a larger share.

Regression losses

The regression losses below are functions of the residual $r = y - \hat{y}$.

Squared error is the loss of least squares, treated in Linear Regression. The two branches of the Huber loss have equal values and slopes at $\lvert r \rvert = \delta$, so the loss is differentiable everywhere. With $\tau = 0.5$, the pinball loss is half the absolute error.

The best constant prediction

The effect of a loss is clearest for a model that predicts one constant $c$ for every sample. Take the labels 2, 3, 4, 5 and 36, where 36 is an outlier. The code tries every constant from 0 to 40 in steps of 0.001 and keeps, for each loss, the one with the smallest average loss.

import numpy as np

y = np.array([2.0, 3.0, 4.0, 5.0, 36.0])
c = np.linspace(0, 40, 40001)       # candidate constants, step 0.001
R = y - c[:, None]                  # residuals: one row per constant
A = np.abs(R)
losses = {
    "squared": R ** 2,
    "absolute": A,
    "Huber, delta=3": np.where(A <= 3, 0.5 * R ** 2, 3 * (A - 1.5)),
    "pinball, tau=0.75": np.maximum(0.75 * R, -0.25 * R),
}
for name, L in losses.items():
    best = c[np.argmin(L.mean(axis=1))]
    print(f"{name}: {best:.3f}")

• Squared error is minimised by the mean, as shown in Decision Trees.
• Absolute error. Where it exists, the derivative of $\sum_i \lvert y_i - c \rvert$ with respect to $c$ is the number of labels below $c$ minus the number above it. Moving $c$ towards the side with more labels therefore lowers the cost, and the minimum is where the two counts balance: at the median.
• Pinball loss. The derivative is $(1 - \tau)$ times the number of labels below $c$ minus $\tau$ times the number above it. It changes sign where about a fraction $\tau$ of the labels lies below $c$, at the $\tau$-quantile of the labels.
• Huber loss lies between the two. As $\delta$ grows from 1 to 40, its minimiser moves from the median, 4, to the mean, 10.

Replacing the outlier 36 by 6 moves the mean from 10 to 4 but leaves the median at 4. Absolute error and the Huber loss are robust: a few extreme labels move their minimiser little. For models with features, squared error trains a model to predict the mean label of samples with similar features, absolute error their median and the pinball loss their $\tau$-quantile, so models trained with $\tau = 0.05$ and $\tau = 0.95$ estimate a 90% prediction interval.

Outliers in a linear model

The labels below follow the line $y = 2x + 1$ with noise, and 10 of the 100 labels are then lowered by 30.

from sklearn.linear_model import (
    HuberRegressor, LinearRegression, QuantileRegressor,
)

rng = np.random.default_rng(0)
x = rng.uniform(0, 10, size=100)
y = 2 * x + 1 + rng.normal(0, 1, size=100)  # true line: y = 2x + 1
bad = rng.choice(100, size=10, replace=False)
y[bad] -= 30                                # corrupt 10 labels
X = x.reshape(-1, 1)

models = [LinearRegression(), HuberRegressor(),
          QuantileRegressor(quantile=0.5, alpha=0)]
for model in models:
    model.fit(X, y)
    print(type(model).__name__, round(model.coef_[0], 3),
          round(model.intercept_, 3))
# LinearRegression 2.241 -3.4
# HuberRegressor 1.99 0.856
# QuantileRegressor 1.973 1.118

To reduce the large squared residuals of the corrupted samples, least squares lowers the whole line, and its intercept falls to −3.4. Both robust fits stay close to the true line. QuantileRegressor adds an L1 penalty with alpha=1.0 by default, so alpha=0 is needed for the pure pinball loss. HuberRegressor applies its threshold epsilon (default 1.35) to residuals divided by a scale that it estimates, so the threshold does not depend on the units of $y$.

Names in scikit-learn and XGBoost

Classification losses

For binary classification, this section codes the labels as $y \in \lbrace -1, +1 \rbrace$. A linear classifier computes a real-valued score $f(x)$, returned by decision_function in scikit-learn, and predicts $+1$ when $f(x) > 0$ and $-1$ otherwise.

A score of 2.5 gives the margin 2.5 for a sample with label $+1$ and $-2.5$ for a sample with label $-1$. Each loss below is a function of the margin alone.

• 0–1 loss: 1 if $m \le 0$ and 0 otherwise; its average is the error rate. It is constant on each side of 0, so its derivative is zero wherever it exists and gives gradient-based training no direction. Training therefore minimises a continuous surrogate loss, such as the three below.
• Log loss: $\log(1 + e^{-m})$. With $\hat{p} = \sigma\bigl(f(x)\bigr)$ and labels coded as 0 and 1, it equals the binary cross-entropy derived in Logistic Regression.
• Hinge loss: $\max(0, 1 - m)$, the loss of the linear support vector machine. It is the default loss of SGDClassifier, and LinearSVC uses it with loss="hinge" (its default is the squared hinge loss). It is 0 for $m \ge 1$, so samples classified correctly with a margin of at least 1 do not affect the fit.
• Exponential loss: $e^{-m}$, the loss minimised by AdaBoost. According to its documentation, GradientBoostingClassifier(loss="exponential") recovers the AdaBoost algorithm.

For large negative margins, the hinge and log losses grow linearly, but the exponential loss is multiplied by $e \approx 2.718$ per unit, so a few badly misclassified samples, such as mislabelled ones, can dominate an exponential cost.

Scores and probabilities

Samples with identical features receive one common score $f$. If a fraction $p$ of them has label $+1$, the code below finds the score that minimises their average loss.

f = np.linspace(-5, 5, 100001)          # candidate scores, step 0.0001
for p in [0.7, 0.9]:
    # average loss when a fraction p of the samples has y = +1
    log = p * np.log(1 + np.exp(-f)) + (1 - p) * np.log(1 + np.exp(f))
    hinge = p * np.maximum(0, 1 - f) + (1 - p) * np.maximum(0, 1 + f)
    f_log, f_hinge = f[np.argmin(log)], f[np.argmin(hinge)]
    p_back = 1 / (1 + np.exp(-f_log))
    print(p, round(f_log, 3), round(p_back, 3), round(f_hinge, 3))
# 0.7 0.847 0.7 1.0
# 0.9 2.197 0.9 1.0

Under log loss, the best score is the log-odds $\log\bigl(p / (1 - p)\bigr)$, 0.847 for $p = 0.7$ and 2.197 for $p = 0.9$, and the sigmoid of that score recovers $p$. Under hinge loss, the best score is 1 in both cases: it records which class is more frequent, but not by how much. A model trained with log loss therefore estimates probabilities, and a model trained with hinge loss only separates the classes.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)
for loss in ["hinge", "log_loss"]:
    clf = make_pipeline(StandardScaler(),
                        SGDClassifier(loss=loss, random_state=0))
    clf.fit(X_train, y_train)
    print(loss, round(clf.score(X_test, y_test), 3),
          hasattr(clf, "predict_proba"))
# hinge 0.965 False
# log_loss 0.965 True

Both models reach a test accuracy of 0.965, but only the log-loss model provides predict_proba. Likewise, the XGBoost documentation states that "binary:hinge" predicts 0 or 1 rather than probabilities, unlike "binary:logistic". For more than two classes, log loss becomes the cross-entropy of multinomial logistic regression, "multi:softprob" in XGBoost.

Losses as likelihoods

Logistic Regression obtains log loss as the negative logarithm of the likelihood, the probability that the model assigns to the observed labels. Squared and absolute error arise in the same way from assumptions about the noise in regression labels. Suppose that each label is the model's prediction plus independent random noise, $y_i = f(x_i) + \varepsilon_i$.

• Gaussian noise. If $\varepsilon_i$ has the normal densityi $\frac{1}{\sigma\sqrt{2\pi}}\, e^{-\varepsilon^2/(2\sigma^2)}$ with standard deviation $\sigma$, the negative log-likelihood of one label is $\bigl(y_i - f(x_i)\bigr)^2 / (2\sigma^2) + \log\bigl(\sigma\sqrt{2\pi}\bigr)$. The second term does not depend on the model, so maximising the likelihood is equivalent to minimising squared error.
• Laplace noise. If $\varepsilon_i$ has the Laplace density $\frac{1}{2b}\, e^{-\lvert \varepsilon \rvert / b}$ with scale $b$, the negative log-likelihood is $\lvert y_i - f(x_i) \rvert / b + \log(2b)$, and maximising the likelihood is equivalent to minimising absolute error.
• Bernoulli labels. If each label is 1 with probability $\hat{p}_i$ and 0 otherwise, the negative log-likelihood is log loss.

A loss is therefore an assumption about the noise. The Laplace density decreases more slowly: for $b = 1$ and $\sigma = 1$, a residual of 4 is $e^{-4} \approx 0.018$ times as probable as a residual of 0 under Laplace noise, but only $e^{-8} \approx 0.00034$ times as probable under Gaussian noise. Absolute error therefore suits labels with occasional large errors.

Weighted losses

In scikit-learn, weights are passed as fit(X, y, sample_weight=w). For LogisticRegression, a weight of 2 gives the same model as including the sample twice:

from sklearn.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)
X = StandardScaler().fit_transform(X)
X_rep = np.vstack([X, X[:1]])         # sample 0 appears twice
y_rep = np.append(y, y[0])
w = np.ones(len(y))
w[0] = 2                              # sample 0 has weight 2

opts = dict(tol=1e-10, max_iter=10_000)
a = LogisticRegression(**opts).fit(X_rep, y_rep)
b = LogisticRegression(**opts).fit(X, y, sample_weight=w)
print(np.allclose(a.coef_, b.coef_),
      np.allclose(a.intercept_, b.intercept_))  # True True

Class weights

Classifiers also accept class_weight, which gives every sample of class $k$ the weight $w_k$, multiplied by its sample_weight if one is given. With class_weight="balanced", scikit-learn uses

$$w_k = \frac{n}{K\, n_k},$$

where $K$ is the number of classes and $n_k$ the number of training samples in class $k$. Every class then has the same total weight, $n / K$. In the training set below, 1,337 samples belong to class 0 and 163 to class 1, so $w_0 = 1500 / (2 \times 1337) \approx 0.561$ and $w_1 = 1500 / (2 \times 163) \approx 4.601$, and both classes have total weight 750.

from sklearn.datasets import make_classification
from sklearn.metrics import precision_score, recall_score
from sklearn.utils.class_weight import compute_class_weight

X, y = make_classification(
    n_samples=2000, n_features=10, n_informative=4, n_redundant=0,
    weights=[0.9], class_sep=1.5, random_state=0,
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)
print(np.bincount(y_train))                      # samples per class
print(compute_class_weight("balanced", classes=np.array([0, 1]),
                           y=y_train).round(3))

for cw in [None, "balanced"]:
    clf = make_pipeline(StandardScaler(),
                        LogisticRegression(class_weight=cw))
    clf.fit(X_train, y_train)
    pred = clf.predict(X_test)
    print(cw, round(recall_score(y_test, pred), 3),
          round(precision_score(y_test, pred), 3))
# [1337  163]
# [0.561 4.601]
# None 0.204 0.846
# balanced 0.63 0.209

Without weights, the model finds only 20.4% of the positive test samples, a recall of 0.204. With balanced weights, the loss of a class-1 sample counts about 8.2 times as much as that of a class-0 sample, and the recall rises to 0.630. The precision, the fraction of positive predictions that are correct, falls from 0.846 to 0.209. Balanced accuracy rises from 0.600 to 0.670 and accuracy falls from 0.910 to 0.702, so whether weighting helps depends on the metric. Lowering the decision threshold of the unweighted model has a similar effect.

Choosing a loss

The training loss should agree with the metric by which the predictions are judged.

Accuracy, F1 and ROC AUC change only in steps as the parameters change, so gradient methods cannot minimise them. A surrogate loss is minimised instead, and the decision threshold is then tuned for the metric, as described in Model Evaluation Metrics.

• Decision trees choose splits by an impurity criterion. The regression criteria "squared_error" and "absolute_error" correspond to the squared and absolute losses, with leaves that predict the mean and the median; the classification criteria are described in Decision Trees.
• Gradient boosting fits each new tree to the negative gradient of the loss, so it works with most losses in this module, as described in XGBoost.
• Neural networks use the same losses, for example as nn.MSELoss, nn.L1Loss and nn.CrossEntropyLoss in PyTorch; the deep learning module on loss functions covers them.