4.1.1Terminology Used in Classical Machine Learning

Task statements, documentation and research papers describe machine learning problems in a shared vocabulary. This module defines the terms used throughout the guide: the parts of a dataset, the main types of learning problem, the components of a model, and the terms used to evaluate one. Structure of a Typical Classical Machine Learning Problem shows how these parts fit together, and Approaching a Typical Classical Machine Learning Problem turns them into a complete workflow.

Samples, features and labels

The data for a classical machine learning problem is usually a table.

The Iris dataset, which is included in scikit-learn, is a standard example. It describes 150 iris flowers by four measurements in centimetres: sepal length, sepal width, petal length and petal width. The label is the species, stored as 0 (setosa), 1 (versicolor) or 2 (virginica). The first two rows and the last row are shown below.

Let us take the first row as an example.

It represents an iris flower with a sepal length and width of 5.1 and 3.5 centimetres, and a petal length and width of 1.4 and 0.2 centimetres. This iris flower, like any other flower in the dataset, is referred to as a "sample." The characteristics of the flower are called "features," and the species of the flower is defined as its "label."

Notation

The following symbols describe any table of this kind. The values in brackets refer to the table above.

• $n$ is the number of samples and $d$ is the number of features ($n = 150$, $d = 4$ in the Iris dataset).
• $x_i$ is the feature vector of sample $i$: its $d$ feature values, in order. For the first flower, $x_1 = (5.1,\ 3.5,\ 1.4,\ 0.2)$.
• $x_{ij}$ is the value of feature $j$ for sample $i$ ($x_{12} = 3.5$).
• $X$ is the feature matrix: all feature values, with one row per sample and one column per feature (150 rows and 4 columns).
• $y$ is the vector of labels, and $y_i$ is the label of sample $i$ ($y_1 = 0$).
• $\hat{y}_i$, read "y-hat", is a model's prediction for sample $i$. The hat distinguishes a prediction from the true label $y_i$.

$$X = \begin{pmatrix} x_{11} & x_{12} & \cdots & x_{1d} \\ x_{21} & x_{22} & \cdots & x_{2d} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \cdots & x_{nd} \end{pmatrix} \qquad y = \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{pmatrix}$$

Row $i$ of $X$ is the feature vector $x_i$, and column $j$ contains feature $j$ for every sample. Since all the values are real numbers, this is often written $x_i \in \mathbb{R}^d$ and $X \in \mathbb{R}^{n \times d}$.

In code, $X$ and $y$ are arrays. scikit-learn expects X to have the shape (n_samples, n_features) and y the shape (n_samples,). Python counts from 0, so sample $i$ is at index $i - 1$.

Several labels per sample

In some problems each sample has more than one target value. These problems are called multi-output. With $m$ target values per sample, the label of sample $i$ is a vector $y_i \in \mathbb{R}^m$, and the labels of all samples form an $n \times m$ matrix $Y$ with one row per sample and one column per target. In Lost in Hyperspace (IOAI 2024), three numeric properties are predicted for each sample, so $m = 3$. scikit-learn expects such labels as y with the shape (n_samples, n_outputs).

Feature engineering and feature extraction

Classical models require each sample to be represented by a feature vector of fixed length. Tabular data already has this form; images, audio and text do not. A 720 × 1280 RGB image, for example, consists of 2,764,800 pixel values, and the meaning of each value depends on its position and its neighbours. Designing features from raw data by hand is called feature engineering. Computing them with a fixed transformation or a pretrained model is called feature extraction.

In Lost in Hyperspace, each sample is a 5 × 5 × 5 × 6 array. The model was fixed to linear regression with at most 300 input features for each predicted property, so the task consisted entirely of feature engineering.

Types of learning problems

The form of the labels determines the type of problem.

• Supervised learning. Every training sample is labelled, and the model learns a mapping from features to labels.
  • Classification: the label takes one of a finite set of values, called classes. A problem with two classes is binary; a problem with more is multi-class. Iris is a multi-class problem with three classes. In Save the Factory (IOAI 2024), each widget belongs to one of two classes, Ruby or Sapphire.
  • Regression: the label is a real number, or several real numbers in a multi-output problem such as Lost in Hyperspace.
• Unsupervised learning. No labels are available. The model finds structure in the features alone, for example groups of similar samples found by K-means clustering.
• Semi-supervised learning. Only some of the training samples are labelled. In Antique Painting Authentication (IOAI 2025), only 4 of the 500 training paintings are labelled, and the statement asks for a model trained on all samples, labelled and unlabelled.

Models, parameters and hyperparameters

A model is a function $f$ that maps a feature vector to a prediction, $\hat{y} = f(x)$. A family of models fixes the form of $f$ and leaves some quantities to be determined. Linear regression, for example, predicts

$$\hat{y} = w^\top x + b = w_1 x_1 + w_2 x_2 + \dots + w_d x_d + b,$$

where the weights $w_1, \dots, w_d$ and the bias $b$ are determined from data.

This might seem complicated, but essentially, a model takes a vector or matrix of numbers as input and spits out a vector or matrix of numbers as output. We will go into this later in CML Models.

Training, or fitting, is the process of choosing parameter values for which the model's predictions agree with the labels of the training data. Agreement is measured by a loss function. For regression, a standard choice is the mean squared error,

$$\mathrm{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left(y_i - \hat{y}_i\right)^2,$$

and Loss Functions covers the common alternatives. Inference, or prediction, is the application of a trained model to new samples.

In scikit-learn, models are called estimators. Every estimator provides fit(X, y) for training and predict(X) for inference.

Save the Factory illustrates the distinction. The organisers fixed both the model, a DecisionTreeClassifier, and its hyperparameters: max_depth=20 for one difficulty level and max_depth=4 for the other. Since neither could be changed, performance depended only on the features supplied to the model.

Training, validation and test sets

A model's performance on its own training data is not a reliable estimate of its performance on new data, because its parameters were chosen to fit those labels. Labelled data is therefore divided into disjoint subsets.

In IOAI tasks, the test labels are withheld and submissions are scored on hidden test sets. Data leakage is any flow of information from validation or test data into training; it makes validation scores unreliable. Approaching a Typical Classical Machine Learning Problem shows how to split data in code and avoid leakage, and Cross-Validation describes how to validate when labelled data is limited.

Evaluation: generalisation, metrics and baselines

Generalisation is a model's ability to make accurate predictions on data it was not trained on.

• Underfitting: the model cannot represent the underlying relationship, so its error is high on both the training data and the validation data.
• Overfitting: the model fits noise specific to the training data, so its training error is low and its validation error is substantially higher.

A metric is the quantity by which predictions are evaluated, such as accuracy or root mean squared error (RMSE). It often differs from the loss used in training. Underfitting and Overfitting and Model Evaluation Metrics treat both topics in detail.

Summary