Khush Shah

Loss Functions for Regression and Classification

Khush Shah 32 min read Machine Learning, Optimisation, Tutorials

Two loss functions can fit clean data equally well and then behave very differently when a few of the targets are wrong, or when one class is much rarer than the other. In this post I implement the common regression losses (MAE, MSE and Huber) and classification losses (binary cross-entropy and focal loss) in a few lines of PyTorch, and train the same linear model with each of them: first on clean data, then on data with 5% outliers and with a 19-to-1 class imbalance, to see when the choice of loss actually matters, and why. Shorter sections cover softmax cross-entropy for more than two classes, the hinge loss, where these losses come from, L1 and L2 penalties, and gradient descent versus Adam, and a table at the end sums up which loss to use when.

Loss

Loss or Objective Function is a measure of the model's performance. It is optimised during the training to improve model's performance. Strictly, the loss is the penalty for a single example and the cost is the mean loss over the data. The formulas below that average over the data are, strictly, costs, but like most people I call both of them the loss.

Broadly speaking, loss functions can be grouped into two major categories concerning the types of problems we come across in the real world: CLASSIFICATION and REGRESSION. In CLASSIFICATION problems, our task is to predict the respective probabilities of all classes the problem is dealing with. When it comes to REGRESSION, our task is to predict the continuous value concerning a given set of independent features to the learning algorithm.

Loss Functions for Regression

Mean Absolute Error Loss

We define MAE loss function as the average of absolute differences between the actual and the predicted value. It measures the average magnitude of errors in a set of predictions, without considering their directions.

$$\mathrm{MAE}(\boldsymbol{y}, \boldsymbol{\hat{y}}) = \frac{1}{n}\sum_{i=1}^{n}|y_i - \hat{y_i}|$$

where $y_i$ is the actual value and $\hat{y_i}$ is the predicted value.

It is also known as the $\ell_1$ loss function.

Mean Squared Error Loss

We define MSE loss function as the average of squared differences between the actual and the predicted value. It’s the most commonly used regression loss function.

$$\mathrm{MSE}(\boldsymbol{y}, \boldsymbol{\hat{y}}) = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y_i})^2$$

where $y_i$ is the actual value and $\hat{y_i}$ is the predicted value.

It is also known as the $\ell_2$ loss function. The MSE loss function penalises the model for making large errors by squaring them.

Huber Loss

We define Huber loss function as a loss that is quadratic for small errors and linear for large ones, so its gradient is bounded by $\delta$. Unlike MAE, which has a kink at 0, it is differentiable everywhere.

$$\mathrm{Huber}(\boldsymbol{y}, \boldsymbol{\hat{y}} ; \delta) = \frac{1}{n}\sum_{i=1}^{n}\mathrm{L}_{\delta}(y_i - \hat{y_i})$$
$$\mathrm{L}_{\delta}(y_i - \hat{y_i}) = \begin{cases} \frac{1}{2}(y_i - \hat{y_i})^2 & \text{for } |y_i - \hat{y_i}| \leq \delta \\ \delta|y_i - \hat{y_i}| - \frac{1}{2}\delta^2 & \mathrm{otherwise} \end{cases}$$

where $y_i$ is the actual value and $\hat{y_i}$ is the predicted value.

The Huber loss function is more robust to outliers compared to the MSE loss function.

Example

Let's take a simple example to understand the above loss functions.

Say for some data, the actual value is 100 and the predicted value is 110, an error of 10. Then the MAE is $10$, the MSE is $10^2 = 100$ and the Huber loss with $\delta = 5$ is $5 \times 10 - \frac{1}{2} \times 5^2 = 37.5$. One value of the loss says little about outliers, though. The first plot shows how the loss of a single example grows with its error $e = \hat{y} - y$, for errors from $-10$ to $10$ (with $\delta = 5$, as in the example):

Show code
import numpy as np
import plotly.graph_objects as go
import plotly.io as pio

pio.renderers.default = "plotly_mimetype+notebook_connected"

delta = 5.0                           # the delta of the example above
e = np.linspace(-10, 10, 401)         # the error of one prediction, e = prediction - actual value

values = {
    "MAE": np.abs(e),
    "MSE": e ** 2,
    "Huber (δ = 5)": np.where(np.abs(e) <= delta, 0.5 * e ** 2, delta * np.abs(e) - 0.5 * delta ** 2),
}
fig = go.Figure()
for name, value in values.items():
    fig.add_scatter(x=e, y=value, mode="lines", name=name)
fig.update_layout(title="Loss against the error",
                  xaxis_title="error e = prediction − actual value", yaxis_title="loss")
fig.show()

The second plot shows the gradient of the same losses with respect to the prediction, $\partial L / \partial \hat{y}$. In gradient descent each example moves the fit in proportion to this gradient, so it measures how hard one example pulls the fitted line towards itself (in robust statistics it is called the influence function):

Show code
gradients = {
    "MAE": np.sign(e),
    "MSE": 2 * e,
    "Huber (δ = 5)": np.clip(e, -delta, delta),
}
fig = go.Figure()
for name, gradient in gradients.items():
    fig.add_scatter(x=e, y=gradient, mode="lines", name=name)
fig.update_layout(title="Gradient against the error",
                  xaxis_title="error e = prediction − actual value", yaxis_title="gradient (pull on the fit)")
fig.show()

The size of the loss is not what makes a loss sensitive to outliers; the gradient is. With MSE the pull of an example grows in proportion to its error, without limit: an example with an error of 10 pulls 10 times as hard as one with an error of 1, so a few large errors can dominate the fit. With MAE every example pulls equally hard, $\pm 1$, whatever the size of its error. With Huber the pull grows linearly up to $\delta$ and then stays at $\pm\delta$, so $\delta$ sets the error beyond which an example counts as an outlier. So the MAE loss function is the least sensitive to outliers, the MSE loss function is the most sensitive, and the Huber loss function sits in between. The experiments below test this.

Implementation

Importing Libraries

The code needs PyTorch, pandas, Plotly and scikit-learn 1.4 or newer, for root_mean_squared_error (the older mean_squared_error(..., squared=False) was deprecated in 1.4 and removed in 1.6).

import numpy as np
import pandas as pd
import plotly.graph_objects as go
import plotly.io as pio
import sklearn
import sklearn.metrics as metrics
import torch
import torch.nn.functional as F
from torch.optim import SGD, Adam

pio.renderers.default = "plotly_mimetype+notebook_connected"
print(f"PyTorch {torch.__version__}, scikit-learn {sklearn.__version__}")
PyTorch 2.8.0, scikit-learn 1.9.0

Defining the loss functions

# MAE loss
def mae(y, y_pred):
    assert y.shape == y_pred.shape
    val = torch.abs(y - y_pred)
    return torch.mean(val)

# MSE loss
def mse(y, y_pred):
    assert y.shape == y_pred.shape
    val = (y - y_pred) ** 2
    return torch.mean(val)

# Huber loss
def huber(y, y_pred, delta=1.0):
    assert y.shape == y_pred.shape
    diff = torch.abs(y - y_pred)
    val = torch.where(diff <= delta, 0.5 * diff ** 2, delta * diff - 0.5 * delta ** 2)
    return torch.mean(val)

Training Function

The model is linear, $\hat{\boldsymbol{y}} = X\boldsymbol{w}$, where the last column of $X$ is all ones so that the last weight is the intercept (or bias). train fits it with Adam on the whole data set at every step, so one step is one epoch, and records the training loss and a metric (RMSE for regression, accuracy for classification) at every epoch. It resets the seed first, so every loss starts from the same random weights. train_all trains one model per loss, and the other two functions plot what they recorded.

def train(x, y, loss_fn, loss_kwargs=None, task="regression", lr=0.05, epochs=2000):
    loss_kwargs = loss_kwargs or {}
    torch.manual_seed(0)                                  # the same starting weights for every loss
    w = torch.randn(x.shape[1], 1, requires_grad=True)
    opt = Adam([w], lr=lr)

    history = {"loss": [], "metric": []}
    for epoch in range(epochs):
        out = torch.matmul(x, w)                          # predictions, or logits for classification
        loss = loss_fn(y, out, **loss_kwargs)
        opt.zero_grad()
        loss.backward()
        opt.step()

        out = out.detach()
        if task == "regression":
            metric = metrics.root_mean_squared_error(y, out)
        else:                                             # the sigmoid is only needed for the accuracy
            y_pred = (torch.sigmoid(out) > 0.5).float()
            metric = metrics.accuracy_score(y, y_pred)
        history["loss"].append(loss.item())
        history["metric"].append(metric)

    return w.detach(), history


def train_all(losses, x, y, **train_kwargs):
    # losses: {name: (loss function, its keyword arguments)}
    weights, histories = {}, {}
    for name, (loss_fn, loss_kwargs) in losses.items():
        weights[name], histories[name] = train(x, y, loss_fn, loss_kwargs, **train_kwargs)
    return weights, histories


def plot_curves(histories, key, title, yaxis_title, log_y=False):
    fig = go.Figure()
    for name, history in histories.items():
        fig.add_scatter(y=history[key], mode="lines", name=name)
    fig.update_layout(title=title, xaxis_title="epoch", yaxis_title=yaxis_title)
    if log_y:
        fig.update_yaxes(type="log")
    fig.show()


def plot_lines(weights, y, title, outliers=None):
    # The fitted lines first, so that every figure gives each loss the same colour.
    fig = go.Figure()
    ends = torch.tensor([[0.0, 1.0], [1.0, 1.0]])        # x = 0 and x = 1, with the column of ones
    for name, w in weights.items():
        fig.add_scatter(x=[0, 1], y=(ends @ w)[:, 0].numpy(), mode="lines", name=name)
    fig.add_scatter(x=[0, 1], y=[3, 5], mode="lines", name="true line, y = 2x + 3", line_dash="dash")
    normal = torch.ones(len(y), dtype=torch.bool)
    if outliers is not None:
        normal[outliers] = False
    fig.add_scatter(x=x[normal, 0].numpy(), y=y[normal, 0].numpy(), mode="markers", name="data",
                    marker=dict(size=5, opacity=0.5))
    if outliers is not None:
        fig.add_scatter(x=x[outliers, 0].numpy(), y=y[outliers, 0].numpy(), mode="markers",
                        name="outliers (+30)", marker=dict(size=6))
    fig.update_layout(title=title, xaxis_title="x", yaxis_title="y")
    fig.show()

Generating the data for regression

I draw 500 points from $y = 2x + 3$ plus Gaussian noise with a standard deviation of $0.5$, and make a second copy of the targets in which 5% of them (25 points) are $30$ too high, as if they had been recorded wrongly. Both experiments use the same $x$ values.

torch.manual_seed(0)
n = 500
x = torch.rand(n, 1)
y_clean = 2 * x + 3 + 0.5 * torch.randn(n, 1)

y_outliers = y_clean.clone()
idx = torch.randperm(n)[: int(0.05 * n)]      # 5% outliers
y_outliers[idx] += 30.0

X = torch.concatenate([x, torch.ones((n, 1))], axis=1)    # a column of ones for the intercept


def line_table(weights):
    # Slope and intercept of each fitted line, how far it is above the true line at the mean x,
    # and its RMSE against the clean targets (the noise alone gives about 0.5).
    rows = {"true line": torch.tensor([[2.0], [3.0]]), **weights}
    return pd.DataFrame({
        name: {"slope": w[0, 0].item(),
               "intercept": w[1, 0].item(),
               "lift at mean x": ((X @ w).mean() - (2 * x + 3).mean()).item(),
               "RMSE on clean targets": metrics.root_mean_squared_error(y_clean, X @ w)}
        for name, w in rows.items()
    }).T.round(3)

Clean data

First, the three losses on the clean data. Each model starts from the same random weights and is trained with Adam for 2,000 epochs at a learning rate of $0.05$:

regression_losses = {
    "MAE": (mae, {}),
    "MSE": (mse, {}),
    "Huber (δ = 1)": (huber, {"delta": 1.0}),
}
clean_weights, clean_histories = train_all(regression_losses, X, y_clean, task="regression")
line_table(clean_weights)
slope intercept lift at mean x RMSE on clean targets
true line 2.000 3.000 0.000 0.530
MAE 1.928 3.006 -0.029 0.530
MSE 1.887 3.062 0.006 0.529
Huber (δ = 1) 1.873 3.068 0.005 0.529
plot_curves(clean_histories, "loss", "Loss per epoch, clean data", "training loss", log_y=True)
plot_curves(clean_histories, "metric", "RMSE per epoch, clean data", "RMSE")
plot_lines(clean_weights, y_clean, "Fitted lines, clean data")

All three losses find nearly the same line, close to $y = 2x + 3$, and all three have an RMSE of 0.53 on the clean targets, the same as the true line itself: what is left is the noise. The loss curves sit at different heights because the three losses measure the error in different ways, so only their shapes can be compared, and the RMSE curves show that all three have converged within about 400 epochs.

Data with outliers

The clean data cannot tell the three losses apart. With Gaussian noise and no outliers, all three estimate the same line, so the first experiment says nothing about the claim that MSE is sensitive to outliers. To test that claim the data needs outliers, so here is the same experiment on the copy of the targets in which 25 of the 500 points were moved up by 30:

outlier_weights, outlier_histories = train_all(regression_losses, X, y_outliers, task="regression")
line_table(outlier_weights)
slope intercept lift at mean x RMSE on clean targets
true line 2.000 3.000 0.000 0.530
MAE 1.951 3.027 0.002 0.529
MSE 3.288 3.869 1.506 1.642
Huber (δ = 1) 1.915 3.106 0.064 0.532
plot_lines(outlier_weights, y_outliers, "Fitted lines, 5% outliers", outliers=idx)

The MAE and Huber lines barely move: their slopes and intercepts stay within 0.11 of $(2, 3)$, and their RMSE on the clean targets is still 0.53. The MSE line is lifted by 1.51 at the mean $x$, which is the $0.05 \times 30 = 1.5$ that the outliers add to the mean of $y$, because a least-squares line always passes through the point $(\bar{x}, \bar{y})$. The 25 outliers happen to sit a little to the right of centre (their mean $x$ is 0.57), so the slope is pulled up too, from 2 to 3.3, and the RMSE on the clean targets triples, to 1.64.

This is what the gradient plot predicted. Sensitivity to outliers is about how much a single large error can move the fit, which is set by the gradient of the loss, not by the size of the loss value. Under MSE each point pulls in proportion to its error, so the line stops moving only when the pull of the 25 outliers, each now about $30 - 1.5 = 28.5$ above it, is balanced by the pull of the 475 ordinary points, each about $1.5$ below it: $25 \times 28.5 \approx 475 \times 1.5$. Under MAE every point pulls with the same force whatever its error, so the 25 outliers count as 25 points above the line and no more, and the line hardly moves. Under Huber with $\delta = 1$ an outlier pulls no harder than an ordinary point with an error of 1.

Loss Functions for Classification

Binary Cross-Entropy Loss

This is the most common loss function used in classification problems. The binary cross-entropy loss decreases as the predicted probability converges to the actual label. It measures the performance of a classification model whose predicted output is a probability value between 0 and 1.

$$\mathrm{L}(\boldsymbol{y}, \boldsymbol{\hat{y}}) = \begin{cases} -\log(\hat{y_i}) & \text{if } y_i = 1 \\ -\log(1-\hat{y_i}) & \text{if } y_i = 0 \end{cases}$$
$$\mathrm{L}(\boldsymbol{y}, \boldsymbol{\hat{y}}) = - \dfrac{1}{m} \sum_{i=1}^{m} y_i \log(\hat{y_i}) + (1-y_i) \log(1-\hat{y_i})$$

where $y_i$ is the actual value and $\hat{y_i}$ is the predicted value.

Focal Loss

Focal loss (Lin et al., 2017, "Focal Loss for Dense Object Detection", the loss behind the RetinaNet detector) multiplies the cross-entropy by a modulating factor $(1 - p_t)^{\gamma}$, where $p_t$ is the probability that the model gives to the true class. The factor is close to 0 for easy, well-classified examples ($p_t$ near 1) and close to 1 for hard ones, so it down-weights the easy examples. Training then concentrates on the hard examples and is less swamped by a large majority class of easy ones. With $\gamma = 0$ it is the cross-entropy again.

$$\mathrm{FL}(\boldsymbol{y}, \boldsymbol{\hat{y}}) = \begin{cases} -(1-\hat{y_i})^{\gamma}\log(\hat{y_i}) & \text{if } y_i = 1 \\ -(\hat{y_i})^{\gamma}\log(1-\hat{y_i}) & \text{if } y_i = 0 \end{cases}$$
$$\mathrm{FL}(\boldsymbol{y}, \boldsymbol{\hat{y}}) = - \dfrac{1}{m} \sum_{i=1}^{m} y_i (1 - \hat{y_i})^{\gamma} \log(\hat{y_i}) + (1-y_i) (\hat{y_i})^{\gamma} \log(1-\hat{y_i})$$

where $y_i$ is the actual label and $\hat{y_i}$ is the predicted probability of the label. With $p_t = \hat{y_i}$ when $y_i = 1$ and $p_t = 1 - \hat{y_i}$ when $y_i = 0$, both cases are one formula, $\mathrm{FL}(p_t) = -(1 - p_t)^{\gamma}\log(p_t)$.

In practice it is used in an $\alpha$-balanced form,

$$\mathrm{FL}(p_t) = -\alpha_t (1 - p_t)^{\gamma}\log(p_t),$$

where $\alpha_t = \alpha$ for class 1 and $\alpha_t = 1 - \alpha$ for class 0. $\alpha$ is a fixed weight for each class, like the class weights of a weighted cross-entropy. The paper found $\gamma = 2$ with $\alpha = 0.25$ to work best, which gives the rare class the smaller weight, because the modulating factor already removes most of the loss of the many easy examples of the common class.

Example

Let's take a simple example to understand the above loss functions.

Say for some data, the actual label is 1 and the predicted probability of the label is 0.85, an easy example. The loss function for the above loss functions will be:

$$\begin{align*} \mathrm{BCE} &= -\log(0.85) \approx 0.1625 \\ \mathrm{FL}(\gamma = 2) &= -(1-0.85)^2\log(0.85) \\ &\approx 0.00366 \end{align*}$$

Now say the predicted probability of the same label is 0.55. The prediction is still correct, because 0.55 is above 0.5, but it is unconfident: a hard, uncertain example. The loss function for the above loss functions will be:

$$\begin{align*} \mathrm{BCE} &= -\log(0.55) \approx 0.5978 \\ \mathrm{FL}(\gamma = 2) &= -(1-0.55)^2\log(0.55) \\ &\approx 0.1211 \end{align*}$$

And if the predicted probability is 0.2, the example is misclassified:

$$\begin{align*} \mathrm{BCE} &= -\log(0.2) \approx 1.609 \\ \mathrm{FL}(\gamma = 2) &= -(1-0.2)^2\log(0.2) \\ &\approx 1.030 \end{align*}$$

Here, we can see that from the easy to the hard example the BCE loss rises 3.7 times ($0.598 / 0.162$), but the FL loss rises about 33 times ($0.121 / 0.00366$). This does not mean that FL penalises the model more: for the same prediction FL is always smaller than BCE, because $(1 - p_t)^{\gamma} < 1$. What changes is the relative weight of hard and easy examples. The loss for an easy example ($p = 0.85$) drops by a factor of 44 relative to BCE, but for a hard one ($p = 0.55$) only by about 5, and for the misclassified one ($p = 0.2$) only by 1.6. In a data set with many easy examples, the hard ones therefore make up a much larger share of the focal loss, and of its gradient, than of the cross-entropy.

Implementation

Defining the loss functions

Both classification losses take the logits $z = X\boldsymbol{w}$, the raw scores before the sigmoid, rather than the probabilities $\hat{y} = \sigma(z)$. binary_cross_entropy_with_logits computes the binary cross-entropy from them without forming the sigmoid. Since the cross-entropy of one example is $-\log p_t$, the focal loss gets $p_t$ back as $e^{-\mathrm{BCE}}$. pos_weight multiplies the loss of every class-1 example, which is the usual way to weight the cross-entropy for imbalanced classes.

# Binary Cross-Entropy loss, from logits
def bce_with_logits(y, logits):
    assert y.shape == logits.shape
    return F.binary_cross_entropy_with_logits(logits, y)

# Class-weighted Binary Cross-Entropy loss, from logits
def weighted_bce_with_logits(y, logits, pos_weight=1.0):
    assert y.shape == logits.shape
    return F.binary_cross_entropy_with_logits(logits, y, pos_weight=torch.tensor(pos_weight))

# Focal loss, from logits
def focal_with_logits(y, logits, gamma=2.0, alpha=None):
    assert y.shape == logits.shape
    bce = F.binary_cross_entropy_with_logits(logits, y, reduction="none")
    p_t = torch.exp(-bce)                       # probability of the true class
    val = (1 - p_t) ** gamma * bce
    if alpha is not None:                       # alpha for class 1, 1 - alpha for class 0
        val = (alpha * y + (1 - alpha) * (1 - y)) * val
    return val.mean()

Why logits? Here is the cross-entropy formula from above, applied to the sigmoid of a confident wrong prediction, a logit of 40 for an example of class 0:

logits = torch.tensor([40.0])
label = torch.tensor([0.0])

prob = torch.sigmoid(logits)
naive = -label * torch.log(prob) - (1 - label) * torch.log(1 - prob)
print(f"sigmoid(40) in float32: {prob.item()}")
print(f"naive BCE:              {naive.item()}")
print(f"BCE from logits:        {bce_with_logits(label, logits).item()}")
sigmoid(40) in float32: 1.0
naive BCE:              inf
BCE from logits:        40.0
A sigmoid followed by a log gives inf

In float32, $\sigma(40) = 1 - 4.2 \times 10^{-18}$ rounds to exactly 1.0, so $1 - \hat{y}$ is 0 and the loss is $-\log 0 = \infty$, although the true loss is about 40. This happens for every logit above about 17 (and, for the other class, below about $-88$, where the sigmoid rounds to 0), and one such example makes the mean loss inf and the gradients nan. From the logit the loss is $y \, \mathrm{softplus}(-z) + (1 - y) \, \mathrm{softplus}(z)$, which can be computed without ever rounding the sigmoid, and that is what binary_cross_entropy_with_logits does. I explain this in more detail in Numerical Hygiene.

Generating the data for classification

The first data set is two balanced clusters of 250 points each. I standardise both features to a mean of 0 and a standard deviation of 1, so that they are on the same scale and the random starting weights give logits of a sensible size (the last section shows why the scale matters).

from sklearn.datasets import make_blobs


def standardise(features, mean, std):
    # Scale each feature to mean 0 and standard deviation 1, and add a column of ones for the bias.
    z = torch.from_numpy((features - mean) / std).float()
    return torch.concatenate([z, torch.ones((len(z), 1))], axis=1)


blob_features, blob_labels = make_blobs(n_samples=500, centers=2, cluster_std=2, random_state=42)
X_blobs = standardise(blob_features, blob_features.mean(axis=0), blob_features.std(axis=0))
y_blobs = torch.from_numpy(blob_labels).float().reshape(-1, 1)


def plot_boundaries(weights, x_cls, y_cls, title, class_names):
    # The decision boundary of each model is the line where the logit is 0 (probability 0.5).
    fig = go.Figure()
    x1 = np.array([x_cls[:, 0].min().item(), x_cls[:, 0].max().item()])
    for name, w in weights.items():
        w = w[:, 0].numpy()
        fig.add_scatter(x=x1, y=-(w[0] * x1 + w[2]) / w[1], mode="lines", name=name)
    for label, marker in ((1, dict(symbol="diamond", size=7, opacity=0.9)), (0, dict(size=5, opacity=0.45))):
        mask = y_cls[:, 0] == label
        fig.add_scatter(x=x_cls[mask, 0].numpy(), y=x_cls[mask, 1].numpy(), mode="markers",
                        name=class_names[label], marker=marker)
    fig.update_layout(title=title, xaxis_title="x₁ (standardised)", yaxis_title="x₂ (standardised)",
                      yaxis_range=[x_cls[:, 1].min().item() - 0.5, x_cls[:, 1].max().item() + 0.5])
    fig.show()

Balanced classes

Both losses on the balanced clusters, trained with Adam for 1,000 epochs at a learning rate of $0.05$:

balanced_losses = {
    "BCE": (bce_with_logits, {}),
    "focal (γ = 2)": (focal_with_logits, {"gamma": 2.0}),
}
blob_weights, blob_histories = train_all(balanced_losses, X_blobs, y_blobs, task="classification", epochs=1000)
pd.DataFrame({name: {"loss at epoch 0": h["loss"][0], "final loss": h["loss"][-1],
                     "accuracy at epoch 0": h["metric"][0], "final accuracy": h["metric"][-1]}
              for name, h in blob_histories.items()}).T.round(4)
loss at epoch 0 final loss accuracy at epoch 0 final accuracy
BCE 0.5522 0.0193 0.594 0.99
focal (γ = 2) 0.2643 0.0048 0.594 0.99
plot_curves(blob_histories, "loss", "Loss per epoch, balanced classes", "training loss", log_y=True)
plot_curves(blob_histories, "metric", "Accuracy per epoch, balanced", "accuracy")
plot_boundaries(blob_weights, X_blobs, y_blobs, "Decision boundaries, balanced",
                class_names={0: "class 0", 1: "class 1"})

Both losses reach 99% accuracy and draw nearly the same boundary. Focal loss has nothing to fix here: both classes have as many easy examples, so down-weighting the easy ones does not change the balance between the classes. The focal loss is lower throughout, as it must be, because $(1 - p_t)^{\gamma} < 1$, but that does not make its model better; as with the regression losses, only the shapes of the loss curves can be compared.

Imbalanced classes

Focal loss was designed for the opposite case, where one class is rare and the other contributes a flood of easy examples. So the second data set has 950 points of class 0 and 50 of class 1, in overlapping clusters. A model that always predicts class 0 is already 95% accurate, so accuracy says little. What matters is how well the model finds the minority class: its precision (the fraction of the points predicted as class 1 that really are class 1), its recall (the fraction of the class-1 points that it finds) and their harmonic mean, $F_1$. I report them on a separate test set of 10,000 points from the same distribution, 500 of them in class 1, so that the numbers are not noisy, and standardise both sets with the mean and standard deviation of the training set.

I train three models: plain BCE, BCE with pos_weight equal to the ratio of the two classes, 19, so that the 50 class-1 points weigh as much as the 950 others, and focal loss with the paper's $\gamma = 2$ and $\alpha = 0.25$.

centers = [[0, 0], [2.5, 2.5]]
imb_features, imb_labels = make_blobs(n_samples=[950, 50], centers=centers, cluster_std=2.0, random_state=42)
test_features, test_labels = make_blobs(n_samples=[9500, 500], centers=centers, cluster_std=2.0, random_state=0)

mean, std = imb_features.mean(axis=0), imb_features.std(axis=0)     # from the training set, used for both
X_imb, X_test = standardise(imb_features, mean, std), standardise(test_features, mean, std)
y_imb = torch.from_numpy(imb_labels).float().reshape(-1, 1)
y_test = torch.from_numpy(test_labels).float().reshape(-1, 1)

pos_weight = (imb_labels == 0).sum() / (imb_labels == 1).sum()
print(f"class 1: {imb_labels.sum()} of {len(imb_labels)} training points, {test_labels.sum()} of {len(test_labels)} test points")
print(f"pos_weight = {pos_weight:g}")
class 1: 50 of 1000 training points, 500 of 10000 test points
pos_weight = 19
imbalanced_losses = {
    "BCE": (bce_with_logits, {}),
    "focal (γ = 2, α = 0.25)": (focal_with_logits, {"gamma": 2.0, "alpha": 0.25}),
    "weighted BCE (pos_weight = 19)": (weighted_bce_with_logits, {"pos_weight": float(pos_weight)}),
}
imb_weights, imb_histories = train_all(imbalanced_losses, X_imb, y_imb, task="classification", epochs=1000)


def minority_table(weights, x_eval, y_eval):
    # Precision, recall and F1 for class 1 on the test set, at the usual threshold of 0.5.
    # Average precision is the area under the precision-recall curve: it uses the scores, not the threshold.
    y_true = y_eval[:, 0].numpy()
    scores = {"always predict class 0": torch.zeros(len(y_true))}
    scores.update({name: (x_eval @ w)[:, 0] for name, w in weights.items()})
    rows = {}
    for name, logit in scores.items():
        y_pred = (torch.sigmoid(logit) > 0.5).numpy()
        precision, recall, f1, _ = metrics.precision_recall_fscore_support(
            y_true, y_pred, average="binary", zero_division=0)
        rows[name] = {"precision": precision, "recall": recall, "F1": f1,
                      "accuracy": metrics.accuracy_score(y_true, y_pred),
                      "average precision": metrics.average_precision_score(y_true, logit.numpy())}
    return pd.DataFrame(rows).T.round(3)


minority_table(imb_weights, X_test, y_test)
precision recall F1 accuracy average precision
always predict class 0 0.000 0.000 0.000 0.950 0.050
BCE 0.701 0.262 0.381 0.958 0.482
focal (γ = 2, α = 0.25) 0.857 0.108 0.192 0.954 0.482
weighted BCE (pos_weight = 19) 0.212 0.792 0.334 0.842 0.482
plot_boundaries(imb_weights, X_imb, y_imb, "Decision boundaries, imbalanced",
                class_names={0: "class 0 (950 points)", 1: "class 1 (50 points)"})

Plain BCE finds only about a quarter of class 1 (recall 0.26), with a precision of 0.70. Focal loss with the paper's settings is worse at the usual threshold of 0.5: it is more precise (0.86) but finds only 11% of class 1, because $\alpha = 0.25$ gives class 1 the smaller weight. In RetinaNet that balances the modulating factor; here it only pushes the boundary further into class 1. Weighted BCE finds 79% of class 1 at the price of many false alarms (precision 0.21), and its $F_1$ of 0.33 is a little below the 0.38 of plain BCE. Accuracy hardly tells the models apart: 0.95 for always predicting class 0, 0.954 and 0.958 for focal loss and BCE, and 0.842 for weighted BCE, the model that finds most of class 1.

The plot shows why the differences are small in a deeper sense. A linear model can only choose a direction and an offset, and the three boundaries are almost parallel: the three losses chose nearly the same direction and mainly moved the boundary. The average precision, which looks at how the scores rank the points and not at the threshold, is 0.482 for all three. So for this model the choice of loss is mostly a choice of threshold, which could just as well be tuned after training with plain BCE. Focal loss matters most where it was invented, in dense object detection with a deep network: a detector scores about 100,000 candidate boxes per image, only a handful of them contain an object, and without the modulating factor the easy background boxes dominate the loss and the gradient.

Comparison between BCE and FL

The figure from the focal-loss paper shows the whole picture: the loss of one example against the probability of its true class, $p_t$, for several values of $\gamma$. I draw it on a logarithmic axis, so that the losses of the easy examples are visible too.

def ce(p):
    return -torch.log(p)

def fl(p, gamma=2):
    return (1 - p) ** gamma * -torch.log(p)
p = torch.arange(0.01, 1, 0.01)          # the probability of the true class, p_t

fig = go.Figure()
for gamma in [0, 0.5, 1, 2, 5]:
    name = "γ = 0 (cross-entropy)" if gamma == 0 else f"γ = {gamma}"
    fig.add_scatter(x=p.numpy(), y=fl(p, gamma).numpy(), mode="lines", name=name)
fig.update_layout(title="Focal loss against pₜ",
                  xaxis_title="probability of the true class, pₜ", yaxis_title="loss")
fig.update_yaxes(type="log", exponentformat="power")
fig.show()

With $\gamma = 0$ the curve is the cross-entropy. As $\gamma$ grows, the right-hand end of the curve bends down: for well-classified examples the loss drops by orders of magnitude (at $p_t = 0.9$, $\gamma = 2$ divides it by 100 and $\gamma = 5$ by 100,000), while for badly classified ones, with $p_t$ near 0, it hardly changes. This is the down-weighting of easy examples in one picture.

Multi-class Cross-Entropy

With $K$ classes the model gives $K$ logits $z_1, \dots, z_K$ for each example, one per class, and the softmax turns them into probabilities that sum to 1:

$$\hat{y}_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}.$$

The cross-entropy of an example of class $c$ is the negative log of the probability of its true class,

$$\mathrm{L} = -\log \hat{y}_c = -z_c + \log \sum_{j=1}^{K} e^{z_j},$$

which is the negative log-likelihood of a categorical label, just as the binary cross-entropy is for a Bernoulli one. In PyTorch it is F.cross_entropy(logits, labels), which takes the logits, of shape (n, K), and the class indices, of shape (n,). Like binary_cross_entropy_with_logits it works out the log-softmax directly, with the log-sum-exp trick, so it must be given the raw logits. Applying the softmax first does not raise an error: F.cross_entropy treats the probabilities as logits, applies a second softmax and silently trains on the wrong loss.

mc_logits = torch.tensor([[2.0, 0.5, -1.0],
                          [0.0, 3.0, 0.0],
                          [200.0, 0.0, -200.0]])     # three examples, three classes
mc_labels = torch.tensor([0, 2, 1])                  # the true class of each example
rows = torch.arange(len(mc_labels))

print("F.cross_entropy:   ", F.cross_entropy(mc_logits, mc_labels, reduction="none"))
print("-log_softmax:      ", -torch.log_softmax(mc_logits, dim=1)[rows, mc_labels])
print("-log(softmax):     ", -torch.log(torch.softmax(mc_logits, dim=1)[rows, mc_labels]))

# With two classes, the logits (0, z) give the binary cross-entropy of the logit z.
z_two = torch.tensor([[1.5], [-0.3]])
y_two = torch.tensor([[1.0], [0.0]])
two_logits = torch.concatenate([torch.zeros_like(z_two), z_two], axis=1)
print("two classes:       ", F.cross_entropy(two_logits, y_two[:, 0].long(), reduction="none"))
print("binary, from logit:", F.binary_cross_entropy_with_logits(z_two, y_two, reduction="none")[:, 0])
F.cross_entropy:    tensor([  0.2413,   3.0949, 200.0000])
-log_softmax:       tensor([  0.2413,   3.0949, 200.0000])
-log(softmax):      tensor([0.2413, 3.0949,    inf])
two classes:        tensor([0.2014, 0.5544])
binary, from logit: tensor([0.2014, 0.5544])

For the first two examples the three versions agree. The third gives its true class a logit 200 below the largest one, so the softmax probability of that class, about $e^{-200}$, underflows to 0 in float32 and the naive version returns inf, while the true loss is 200. The last two lines show that with two classes the softmax cross-entropy is the binary cross-entropy of the difference of the two logits, $z_1 - z_0$, because $e^{z_1} / (e^{z_0} + e^{z_1}) = \sigma(z_1 - z_0)$. The binary losses above are the case $K = 2$.

Hinge Loss

The hinge loss is the loss of the support vector machine (Cortes and Vapnik, 1995). It uses the labels $s_i = \pm 1$ instead of 0 and 1, and the margin $m_i = s_i z_i$ of each example, which is positive when the example is on the correct side of the boundary and grows the further it is from it:

$$\mathrm{L}(m_i) = \max(0, 1 - m_i).$$

An example with a margin of at least 1 has zero loss and zero gradient, so once the model classifies it correctly with some room to spare, it stops pulling on the boundary; only the examples near the boundary or on the wrong side of it (the support vectors) shape it. Written in terms of the margin, the binary cross-entropy is $\log(1 + e^{-m_i})$, which never reaches zero: it keeps pushing well-classified examples further out, only more and more weakly.

# Hinge loss, from logits, with the labels 0 and 1 mapped to -1 and +1
def hinge_with_logits(y, logits):
    assert y.shape == logits.shape
    margin = (2 * y - 1) * logits
    return torch.clamp(1 - margin, min=0).mean()
Show code
margins = np.linspace(-3, 3, 241)
fig = go.Figure()
fig.add_scatter(x=margins, y=np.maximum(0, 1 - margins), mode="lines", name="hinge")
fig.add_scatter(x=margins, y=np.log1p(np.exp(-margins)), mode="lines", name="binary cross-entropy")
fig.update_layout(title="Loss against the margin",
                  xaxis_title="margin m = s · z", yaxis_title="loss")
fig.show()

Here is the hinge loss on the balanced clusters, trained like the BCE model above, from the same starting weights for 1,000 epochs:

hinge_weights, hinge_histories = train_all({"hinge": (hinge_with_logits, {})}, X_blobs, y_blobs,
                                           task="classification", epochs=1000)
hinge_margins = ((2 * y_blobs - 1) * (X_blobs @ hinge_weights["hinge"]))[:, 0]
print(f"final accuracy: hinge {hinge_histories['hinge']['metric'][-1]:.3f}, "
      f"BCE {blob_histories['BCE']['metric'][-1]:.3f}")
print(f"points with a non-zero hinge loss at the end: {int((hinge_margins < 1).sum())} of {len(hinge_margins)}")
final accuracy: hinge 0.992, BCE 0.990
points with a non-zero hinge loss at the end: 10 of 500

For the decision the hinge loss does as well as BCE, 99.2% accuracy against 99%, and at the end only 10 of the 500 points have a non-zero loss, so the other 490 no longer affect the boundary at all. The price is that its scores are not probabilities. The hinge loss does not care whether a margin is 1 or 10, so nothing makes $\sigma(z)$ of a hinge model match how often the class occurs; when a probability is needed, use the cross-entropy. An SVM also adds an L2 penalty on the weights, which makes it prefer a boundary with a wide margin; penalties like that are the subject of Regularised Losses below.

Where Do These Losses Come From?

These losses are not arbitrary choices. Each one is the negative log-likelihood of a probability model for the targets, so minimising it is maximum likelihood estimation under that model (Goodfellow et al., 2016, Section 5.5).

MSE and Gaussian noise. If $y_i = \hat{y}_i + \varepsilon_i$ with $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$, then

$$p(y_i \mid \hat{y}_i) = \frac{1}{\sqrt{2\pi}\,\sigma} e^{-(y_i - \hat{y}_i)^2 / 2\sigma^2},$$

and the negative log-likelihood of $n$ independent examples is

$$\begin{aligned} -\log p(\boldsymbol{y} \mid \boldsymbol{\hat{y}}) &= \frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2 \\ &\quad + \frac{n}{2}\log(2\pi\sigma^2). \end{aligned}$$

For a fixed $\sigma$ the second term is a constant, so minimising the MSE maximises the Gaussian likelihood. Gaussian noise is the natural default when the error is the sum of many small independent effects, because the Central Limit Theorem says that such a sum is approximately Gaussian.

MAE and Laplace noise. If instead the noise has the Laplace density $p(\varepsilon) = \frac{1}{2b} e^{-|\varepsilon|/b}$, the same steps give

$$\begin{aligned} -\log p(\boldsymbol{y} \mid \boldsymbol{\hat{y}}) &= \frac{1}{b}\sum_{i=1}^{n}|y_i - \hat{y}_i| \\ &\quad + n\log(2b), \end{aligned}$$

so minimising the MAE is maximum likelihood for Laplace noise. The Laplace density has much heavier tails than the Gaussian, so a large error is far less surprising under it, which is another way to see why MAE is less disturbed by outliers. It also explains the outlier experiment: the constant that minimises the MSE of a set of numbers is their mean, the one that minimises the MAE is their median, and 5% of outliers move the mean a long way but the median hardly at all. Huber (1964) derived his loss for exactly the situation in that experiment, Gaussian noise with a small fraction of gross errors; it is the negative log-likelihood of a density that is Gaussian in the middle and has Laplace-like tails.

Cross-entropy and Bernoulli labels. For a binary label with $P(y_i = 1) = \hat{y}_i$, the likelihood of one example is $\hat{y}_i^{\,y_i}(1 - \hat{y}_i)^{1 - y_i}$, and the negative log-likelihood of the data set,

$$\begin{aligned} -\sum_{i=1}^{m}\Bigl[&y_i \log \hat{y}_i \\ &+ (1 - y_i)\log(1 - \hat{y}_i)\Bigr], \end{aligned}$$

is $m$ times the binary cross-entropy. This is also where the KL-Divergence comes in. For the distribution $p$ of the labels and the distribution $q$ that the model predicts,

$$H(p, q) = H(p) + D_{\mathrm{KL}}(p \,\|\, q),$$

and the entropy $H(p)$ of the labels does not depend on the model, so minimising the cross-entropy is the same as minimising the KL divergence $D_{\mathrm{KL}}(p \,\|\, q)$.

Regularised Losses

When there are many features and few examples, a model can fit the noise in the training data as well as the signal. The usual fix is to add a penalty on the size of the weights to the loss, either the L2 penalty

$$\mathrm{MSE}(\boldsymbol{y}, \boldsymbol{\hat{y}}) + \lambda \sum_{j} w_j^2$$

or the L1 penalty

$$\mathrm{MSE}(\boldsymbol{y}, \boldsymbol{\hat{y}}) + \lambda \sum_{j} |w_j|,$$

where the sums run over the weights of the features but not the intercept, and $\lambda \geq 0$ sets how much the penalty counts. With MSE they are ridge regression (L2, Hoerl and Kennard, 1970) and the lasso (L1, Tibshirani, 1996), but the same penalties can be added to any loss in this post.

They also come from a probability model. If the Gaussian noise model of the last section is given a Gaussian prior on the weights, $w_j \sim \mathcal{N}(0, \tau^2)$, and the weights are chosen to maximise the posterior instead of the likelihood (maximum a posteriori, or MAP, estimation), the loss is the MSE plus the L2 penalty with $\lambda = \sigma^2 / (n\tau^2)$. A Laplace prior gives the L1 penalty in the same way (Goodfellow et al., 2016, Sections 5.6.1 and 7.1).

The gradients of the two penalties differ in the same way as those of MSE and MAE in the second plot of this post. The gradient of $\lambda w_j^2$ is $2\lambda w_j$, which shrinks as the weight shrinks, so the L2 penalty makes every weight smaller but rarely exactly zero. The gradient of $\lambda |w_j|$ is $\pm\lambda$ whatever the size of the weight, so the L1 penalty keeps pushing a small weight with the same force until it reaches zero, and keeps it there as long as the pull of the data on it is weaker than $\lambda$. That is why the lasso sets the weights of unhelpful features to zero and is used to select features.

To see both at work, I make a regression problem with 20 features and only 40 training examples, with Gaussian noise of standard deviation 0.5 as before, in two versions: a sparse one in which only 3 of the 20 features matter (weights 3, −2 and 1.5; the other 17 are zero) and a dense one in which all 20 matter a little (weights drawn from $\mathcal{N}(0, 0.5^2)$). I fit each with MSE alone and with each penalty at four values of $\lambda$, using Adam at a learning rate of 0.01 for 2,000 epochs, and measure the RMSE on 2,000 test examples. In practice $\lambda$ would be chosen by cross-validation; the table shows the test RMSE for a range of values to show the trade-off.

torch.manual_seed(0)
n_train, n_test, d = 40, 2000, 20
true_weights = {"sparse": torch.zeros(d, 1), "dense": 0.5 * torch.randn(d, 1)}
true_weights["sparse"][:3, 0] = torch.tensor([3.0, -2.0, 1.5])
feat_train, feat_test = torch.randn(n_train, d), torch.randn(n_test, d)
noise_train, noise_test = 0.5 * torch.randn(n_train, 1), 0.5 * torch.randn(n_test, 1)
X_pen_train = torch.concatenate([feat_train, torch.ones((n_train, 1))], axis=1)
X_pen_test = torch.concatenate([feat_test, torch.ones((n_test, 1))], axis=1)


def fit_penalised(x, y, penalty=None, lam=0.0, lr=0.01, epochs=2000):
    # MSE plus an L1 or L2 penalty on every weight except the last one, the intercept.
    torch.manual_seed(0)
    w = torch.randn(x.shape[1], 1, requires_grad=True)
    opt = Adam([w], lr=lr)
    for epoch in range(epochs):
        loss = mse(y, x @ w)
        if penalty == "L2":
            loss = loss + lam * (w[:-1] ** 2).sum()
        elif penalty == "L1":
            loss = loss + lam * w[:-1].abs().sum()
        opt.zero_grad()
        loss.backward()
        opt.step()
    return w.detach()


fits = [(None, 0.0)] + [(penalty, lam) for penalty in ("L2", "L1") for lam in (0.01, 0.03, 0.1, 0.3)]
pen_targets, pen_weights, pen_table = {}, {}, {}
for truth, w_true in true_weights.items():
    y_train = feat_train @ w_true + 1.0 + noise_train
    y_test_pen = feat_test @ w_true + 1.0 + noise_test
    pen_targets[truth] = (y_train, y_test_pen)
    for penalty, lam in fits:
        w = fit_penalised(X_pen_train, y_train, penalty, lam)
        pen_weights[truth, penalty, lam] = w
        row = "MSE alone" if penalty is None else f"MSE + {penalty}, λ = {lam:g}"
        pen_table.setdefault(row, {})
        pen_table[row][f"{truth}: test RMSE"] = metrics.root_mean_squared_error(y_test_pen, X_pen_test @ w)
        pen_table[row][f"{truth}: near 0"] = int((w[:-1].abs() < 0.01).sum())

for truth, (y_train, _) in pen_targets.items():
    train_rmse = metrics.root_mean_squared_error(y_train, X_pen_train @ pen_weights[truth, None, 0.0])
    print(f"{truth}: training RMSE of MSE alone {train_rmse:.3f}")
pen_frame = pd.DataFrame(pen_table).T.round(3)
counts = [column for column in pen_frame if column.endswith("near 0")]
pen_frame.astype({column: int for column in counts})
sparse: training RMSE of MSE alone 0.379
dense: training RMSE of MSE alone 0.379
sparse: test RMSE sparse: near 0 dense: test RMSE dense: near 0
MSE alone 0.670 1 0.670 0
MSE + L2, λ = 0.01 0.685 0 0.649 1
MSE + L2, λ = 0.03 0.746 0 0.626 0
MSE + L2, λ = 0.1 1.052 1 0.662 0
MSE + L2, λ = 0.3 1.707 1 0.961 0
MSE + L1, λ = 0.01 0.640 3 0.655 1
MSE + L1, λ = 0.03 0.601 4 0.640 1
MSE + L1, λ = 0.1 0.596 11 0.700 2
MSE + L1, λ = 0.3 0.715 16 1.021 3
from sklearn.linear_model import Lasso

# scikit-learn's Lasso minimises MSE / 2 + alpha * sum |w_j|, so alpha = λ / 2 is the same problem.
y_train, y_test_pen = pen_targets["sparse"]
lasso = Lasso(alpha=0.1 / 2).fit(feat_train.numpy(), y_train[:, 0].numpy())
w_adam = pen_weights["sparse", "L1", 0.1][:-1, 0].numpy()
print(f"lasso by coordinate descent, λ = 0.1: {(lasso.coef_ == 0).sum()} of {d} weights exactly 0, "
      f"test RMSE {metrics.root_mean_squared_error(y_test_pen[:, 0].numpy(), lasso.predict(feat_test.numpy())):.3f}")
print(f"largest difference from the weights found by Adam: {np.abs(lasso.coef_ - w_adam).max():.3f}")
lasso by coordinate descent, λ = 0.1: 10 of 20 weights exactly 0, test RMSE 0.595
largest difference from the weights found by Adam: 0.013

Without a penalty the training RMSE is 0.379 in both versions, below the noise level of 0.5, while the test RMSE is 0.670: with 21 weights for 40 examples the model fits part of the noise. (The two training RMSEs are equal because the residuals of a least-squares fit depend only on the noise, not on the true weights.) The columns "near 0" count the weights below 0.01 in size.

On the sparse problem the L1 penalty helps and the L2 penalty does not. At $\lambda = 0.1$ the L1 penalty brings the test RMSE down from 0.670 to 0.596 and puts 11 of the 20 weights near 0, while every L2 penalty makes the test RMSE worse, because shrinking the three large true weights costs more than shrinking the 17 others saves. On the dense problem it is the other way round: the best L2 penalty, at $\lambda = 0.03$, gives 0.626, and the best L1 penalty only 0.640, because there are no features to drop. This is the rule of thumb from Tibshirani's paper: L1 when a few features matter, L2 when many matter a little. A $\lambda$ that is too large hurts both, because it shrinks the useful weights as well.

Adam does not set any weight exactly to zero: the gradient of $|w_j|$ flips sign as the weight crosses zero, so a gradient method leaves it hovering close to zero rather than at it. scikit-learn's Lasso, which uses coordinate descent, solves the same problem exactly. At $\lambda = 0.1$ it sets 10 of the 20 weights to exactly 0, with a test RMSE of 0.595, and its weights differ from Adam's by at most 0.013.

In PyTorch an L2 penalty is often set through the optimiser's weight_decay argument instead. It adds $\lambda' \boldsymbol{w}$ to the gradient, where $\lambda'$ is the weight_decay value, for every parameter given to the optimiser, the intercept included unless the parameters are split into groups. That is the gradient of $\frac{\lambda'}{2}\lVert\boldsymbol{w}\rVert^2$, so for both SGD and Adam it is an L2 penalty. With Adam, though, the gradient of the penalty is rescaled with the rest of the gradient, so weights with large gradients are shrunk less. AdamW (Loshchilov and Hutter, 2019) instead shrinks every weight by the same fraction at each step, outside the rescaling, which the authors found to generalise better.

Gradient Descent versus Adam

Every model in this post was trained with Adam, and I have not said why. Here is the simplest comparison: plain gradient descent, which moves each weight by the learning rate times its gradient, against Adam, both on the clean regression data with MSE. To show the difference that matters in practice, I train both on two versions of the same data: once with $x$ as it is, in $[0, 1]$, and once with $x$ multiplied by 100, as if it were measured in units 100 times smaller (centimetres instead of metres). The model fits both equally well; only the slope becomes 100 times smaller. Each optimiser keeps the same learning rate on both versions.

def descend(x_opt, y_opt, optimiser, lr, epochs=2000):
    # MSE on the full data set; stops early if the loss is no longer finite.
    torch.manual_seed(0)
    w = torch.randn(x_opt.shape[1], 1, requires_grad=True)
    opt = optimiser([w], lr=lr)
    losses = []
    for epoch in range(epochs):
        loss = mse(y_opt, x_opt @ w)
        losses.append(loss.item())
        if not torch.isfinite(loss):
            break
        opt.zero_grad()
        loss.backward()
        opt.step()
    return w.detach(), losses


X_scaled = torch.concatenate([100 * x, torch.ones((n, 1))], axis=1)

# Gradient descent on MSE diverges when lr > 2 / (largest eigenvalue of the Hessian, 2 X^T X / n).
for name, data in (("x in [0, 1]", X), ("x in [0, 100]", X_scaled)):
    largest = torch.linalg.eigvalsh(2 * data.T @ data / n).max().item()
    print(f"{name}: gradient descent is stable for lr < {2 / largest:.2g}")

runs = {
    "GD, lr 0.1, x in [0, 1]": (X, SGD, 0.1, 1),
    "Adam, lr 0.05, x in [0, 1]": (X, Adam, 0.05, 1),
    "GD, lr 0.1, x in [0, 100]": (X_scaled, SGD, 0.1, 100),
    "Adam, lr 0.05, x in [0, 100]": (X_scaled, Adam, 0.05, 100),
    "GD, lr 0.00025, x in [0, 100]": (X_scaled, SGD, 2.5e-4, 100),
}
results, curves = {}, {}
for name, (data, optimiser, lr, scale) in runs.items():
    w, losses = descend(data, y_clean, optimiser, lr)
    curves[name] = losses
    diverged = not np.isfinite(losses[-1])
    results[name] = {"epochs run": len(losses) - diverged, "final loss": losses[-1],
                     "slope (per unit of the original x)": scale * w[0, 0].item(), "intercept": w[1, 0].item()}
pd.DataFrame(results).T.map("{:.4g}".format)
x in [0, 1]: gradient descent is stable for lr < 0.79
x in [0, 100]: gradient descent is stable for lr < 0.0003
epochs run final loss slope (per unit of the original x) intercept
GD, lr 0.1, x in [0, 1] 2000 0.2797 1.887 3.062
Adam, lr 0.05, x in [0, 1] 2000 0.2797 1.887 3.062
GD, lr 0.1, x in [0, 100] 6 inf 1.208e+19 1.811e+15
Adam, lr 0.05, x in [0, 100] 2000 0.2802 1.957 3.016
GD, lr 0.00025, x in [0, 100] 2000 2.038 5.799 0.453
def plot_descent(scale):
    # One figure per version of the data; gradient descent and Adam keep their colours in both.
    fig = go.Figure()
    for name, losses in curves.items():
        if name.endswith(scale):
            finite = [v for v in losses if np.isfinite(v)]
            fig.add_scatter(x=np.arange(1, len(finite) + 1), y=finite, mode="lines",
                            name=name.removesuffix(", " + scale))
    fig.update_layout(title=f"MSE per epoch, {scale}", xaxis_title="epoch (log scale)",
                      yaxis_title="training loss (MSE)")
    fig.update_xaxes(type="log")
    fig.update_yaxes(type="log", range=[-1, 4.5], exponentformat="power")
    fig.show()
plot_descent("x in [0, 1]")
plot_descent("x in [0, 100]")

On $x$ in $[0, 1]$ both optimisers reach the least-squares line (slope 1.887, intercept 3.062, the MSE fit from the clean-data experiment), and gradient descent even gets there first. Multiplying $x$ by 100 changes everything for gradient descent. The gradient for the slope becomes about 100 times larger and the curvature of the loss along the slope 10,000 times larger, so the learning rate that worked before now overshoots further at every step, and the weights overflow to inf after 6 epochs. On this data gradient descent is only stable for a learning rate below $3 \times 10^{-4}$, and at $2.5 \times 10^{-4}$ the intercept, whose gradient did not grow, has only reached 0.45 after 2,000 epochs. Adam converges on both versions with the same learning rate, because it divides the gradient of each weight by a running estimate of its typical size (the square root of a moving average of its square), so that each weight moves by roughly the learning rate per step whatever the scale of its gradient. It is not immune: on the scaled data it needs about 1,600 epochs instead of 350, because the two weights are still coupled. Standardising the features, as I did for the classification data, is the usual fix, and it helps both optimisers.

Choosing a Loss

The table at the end of this section sums up what the experiments in this post suggest. One row needs a check that I have not made yet: whether the predicted probabilities can be trusted. A model is calibrated when, of the examples to which it gives a probability of about 0.2, about 20% are in class 1. On the imbalanced test set, where 5% of the points are in class 1, a simple check is the mean predicted probability, which a calibrated model should get close to 0.05, together with the log loss (the binary cross-entropy on the test set) and the Brier score (the mean squared difference between the probability and the label), both lower for better probabilities:

y_true = y_test[:, 0].numpy()
calibration = {}
for name, w in imb_weights.items():
    prob = torch.sigmoid(X_test @ w)[:, 0].numpy()
    calibration[name] = {"mean probability": prob.mean(),
                         "median for class 0": np.median(prob[y_true == 0]),
                         "log loss": metrics.log_loss(y_true, prob),
                         "Brier score": metrics.brier_score_loss(y_true, prob)}
pd.DataFrame(calibration).T.round(4)
mean probability median for class 0 log loss Brier score
BCE 0.0465 0.0073 0.1280 0.0342
focal (γ = 2, α = 0.25) 0.1254 0.0977 0.1849 0.0427
weighted BCE (pos_weight = 19) 0.2593 0.1402 0.3600 0.1131

Plain BCE gives a mean probability of 0.047, close to the true 5%, and the lowest log loss and Brier score. Weighted BCE gives a mean of 0.26, five times too high, because pos_weight = 19 trains the model as if class 1 were 19 times as common as it is. Focal loss gives 0.13, although it is the model that predicts class 1 least often: by down-weighting the easy examples it stops pushing their probabilities towards 0, so a typical class-0 point gets a probability of 0.098 instead of BCE's 0.007. So the two losses that help the minority class do it by distorting the probabilities. When the probabilities matter, train with plain BCE, and if the minority class must be found, lower the threshold instead.

Plain cross-entropy is calibrated here because its expected value is smallest when the predicted probability is the true one, and because a linear model with three weights cannot overfit 1,000 points. Large neural networks trained with cross-entropy are often over-confident; the usual fix is to divide their logits by a constant fitted on held-out data after training (temperature scaling, Guo et al., 2017).

Situation Loss What this post found
Regression, noise roughly Gaussian MSE On the clean data MAE, MSE and Huber fit the same line; MSE is the maximum-likelihood loss for Gaussian noise.
Regression with outliers in the targets Huber or MAE With 5% outliers the MSE line was lifted by 1.51, while MAE and Huber stayed within 0.11 of $(2, 3)$.
Classification, calibrated probabilities needed plain BCE, from logits Mean probability 0.047 for a true rate of 0.05, and the lowest log loss and Brier score of the three.
Imbalanced classes, the minority class must be found weighted BCE, or plain BCE with a lower threshold Weighted BCE raised the minority recall from 0.26 to 0.79. All three models ranked the points equally well (average precision 0.482), so the threshold of plain BCE can do much the same job.
Extreme imbalance with a deep network, as in dense object detection focal loss, with $\gamma$ and $\alpha$ tuned Not tested here. With a linear model and the paper's $\gamma = 2$, $\alpha = 0.25$ it did worst of the three: recall 0.11, $F_1$ 0.19.
More than two classes softmax cross-entropy, F.cross_entropy on logits It is the binary cross-entropy when $K = 2$.
Only the decision matters, not the probability hinge 99.2% accuracy on the balanced clusters, against 99% for BCE, but its scores are not probabilities.
Many features, few examples add an L2 penalty if many features matter a little, L1 if only a few do With 20 features and 40 examples, L1 cut the test RMSE from 0.670 to 0.596 on the sparse problem, and L2 to 0.626 on the dense one.

Takeaways

  • On clean data MAE, MSE and Huber fit the same line. With 5% outliers the MSE line is lifted by 1.5 ($0.05 \times 30$) and its RMSE on the clean targets triples, while MAE and Huber stay within 0.11 of $(2, 3)$. What decides this is the gradient, not the size of the loss: under MSE a point pulls on the fit in proportion to its error, while under MAE and Huber its pull is capped at 1 and $\delta$.
  • Compute the cross-entropy and the focal loss from logits with binary_cross_entropy_with_logits, because a sigmoid followed by a log returns inf for one confident mistake. Each of these losses is a negative log-likelihood: MSE for Gaussian noise, MAE for Laplace noise and cross-entropy for Bernoulli labels.
  • Focal loss is always smaller than the cross-entropy; what it changes is the relative weight of easy and hard examples. On imbalanced data judge a classifier by the precision, recall and $F_1$ of the minority class, not by accuracy, which always predicting the majority class already gets to 95%. For a linear model, plain, weighted and focal cross-entropy ranked the points almost identically (average precision 0.482) and differed mainly in where they put the threshold.

References

  1. P. J. Huber, "Robust Estimation of a Location Parameter", The Annals of Mathematical Statistics 35(1), 73–101, 1964.
  2. T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, "Focal Loss for Dense Object Detection", ICCV, 2017. arXiv:1708.02002.
  3. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016. Chapter 5, Section 5.5, covers maximum likelihood and its link to MSE and cross-entropy; Sections 5.6.1 and 7.1 link Gaussian and Laplace priors to the L2 and L1 penalties.
  4. D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization", ICLR, 2015. arXiv:1412.6980.
  5. PyTorch documentation: torch.nn.functional.binary_cross_entropy_with_logits and torch.nn.functional.cross_entropy.
  6. scikit-learn documentation: sklearn.metrics.root_mean_squared_error, sklearn.metrics.precision_recall_fscore_support, sklearn.metrics.average_precision_score, sklearn.metrics.log_loss, sklearn.metrics.brier_score_loss and sklearn.linear_model.Lasso.
  7. C. Cortes and V. Vapnik, "Support-Vector Networks", Machine Learning 20(3), 273–297, 1995.
  8. A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics 12(1), 55–67, 1970.
  9. R. Tibshirani, "Regression Shrinkage and Selection via the Lasso", Journal of the Royal Statistical Society, Series B 58(1), 267–288, 1996.
  10. I. Loshchilov and F. Hutter, "Decoupled Weight Decay Regularization", ICLR, 2019. arXiv:1711.05101.
  11. C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, "On Calibration of Modern Neural Networks", ICML, 2017. arXiv:1706.04599.