4.1.3Approaching a Typical Classical Machine Learning Problem

Structure of a Typical Classical Machine Learning Problem broke a problem into four parts: the input and target, preprocessing, model training, and prediction with evaluation. That module answered the question of what a problem is made of. This module answers a different one: what do you actually do, step by step, when a new task appears in front of you?

The answer matters more than it might seem. In a contest, a strong model can still score nothing because validation data leaked into training, because the model was judged by the wrong metric, or because the submission file had an extra column. None of these mistakes has anything to do with how good the model is, and all of them are easy to avoid with a fixed routine.

This module builds that routine on one complete example, set up in the same way as a contest task, from reading the statement to checking the submission file. You will use the same routine on every task in this guide, whether the final model is a linear regression, a random forest or a neural network.

1. Read the statement: its inputs, target, metric, submission format and constraints.
2. Load the data and inspect its shape, column types, missing values and labels.
3. Split the labelled data into a training set and a validation set.
4. Fit every preprocessing step on the training set only.
5. Evaluate a baseline and a first model on the validation set, using the task's metric.
6. Improve the features, models and hyperparameters, evaluating each change as in step 5, while time allows.
7. Refit the chosen model on all labelled data and predict the test inputs.
8. Write the submission file and check it against the required format.

Reading the task statement

A task statement contains more than the problem description. Before any code is written, five facts should be extracted from it, since each one constrains a later step.

The example task

The rest of this module solves a small task built from the Breast Cancer dataset, which is included in scikit-learn. Each of its 569 samples describes a breast tissue sample by 30 numeric features computed from a microscope image of cell nuclei. The label is 0 for a malignant tumour and 1 for a benign one.

The first code block plays the role of the organisers. It keeps 20% of the samples as a test set, removes their labels, and stores the labels separately for grading.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

# Organiser side: build the task files from a bundled dataset
df = load_breast_cancer(as_frame=True).frame
df.insert(0, "id", range(1, len(df) + 1))
train_df, test_df = train_test_split(
    df, test_size=0.2, stratify=df["target"], random_state=0
)
train_df.to_csv("train.csv", index=False)
test_df.drop(columns="target").to_csv("test.csv", index=False)
test_df[["id", "target"]].to_csv("hidden_labels.csv", index=False)

The statement for the task reads as follows.

Balanced accuracy is the mean, over the classes, of the fraction of samples of that class that are predicted correctly:

$$\text{balanced accuracy} = \frac{1}{2}\left(\frac{\text{malignant samples predicted malignant}}{\text{malignant samples}} + \frac{\text{benign samples predicted benign}}{\text{benign samples}}\right).$$

Unlike plain accuracy, it cannot be raised by predicting the more common class for every sample. Model Evaluation Metrics covers it together with the other common metrics.

Loading and inspecting the data

The first inspection answers four questions: how much data there is, how the training and test files differ, what types the columns have, and whether any values are missing.

import pandas as pd

train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

print(train.shape, test.shape)                    # (455, 32) (114, 31)
print(set(train.columns) - set(test.columns))     # {'target'}
print(train.dtypes.astype(str).value_counts().to_dict())
# {'float64': 30, 'int64': 2}
print(train.isna().sum().sum())                   # 0
print(train["target"].value_counts().to_dict())   # {1: 285, 0: 170}

The training file has 455 labelled samples and the test file 114 unlabelled ones. The only column missing from the test file is target. The 30 features are floating-point numbers, the two integer columns are id and target, and no values are missing. The classes are imbalanced: 285 benign samples against 170 malignant ones, which is why the statement uses balanced accuracy. Real datasets are rarely this clean; Pandas and Data Cleaning cover how to inspect and repair them.

Splitting the labelled data

The test labels are hidden, so the only way to estimate test performance before submitting is to hold back part of the labelled data as a validation set. The id column is excluded from the features: it identifies a row but carries no information about the label.

features = [c for c in train.columns if c not in ("id", "target")]
X = train[features]
y = train["target"]

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)
print(len(X_train), len(X_val))          # 364 91
print(y_val.value_counts().to_dict())    # {1: 57, 0: 34}

Two arguments matter here.

• stratify=y keeps the class proportions the same in both parts. Without it, a small validation set can by chance contain very few samples of the rarer class.
• random_state=0 fixes the random shuffle, so the notebook produces the same split every time it runs.

Preprocessing without leakage

Many preprocessing steps learn values from data. StandardScaler, for example, learns the mean and standard deviation of each feature and uses them to rescale it. Such a step must be fitted on the training split only and then applied, unchanged, to the validation and test data.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # learns mean and std
X_val_scaled = scaler.transform(X_val)          # reuses them

print(round(float(X_train_scaled[:, 0].mean()), 3))  # 0.0
print(round(float(X_val_scaled[:, 0].mean()), 3))    # -0.045

The first feature has mean exactly 0 on the training split, where the scaler was fitted, and a mean close to, but not equal to, 0 on the validation split. This small difference is expected: the validation data plays the part of new data, which the scaler has never seen.

Fitting each step by hand on the right subset becomes error-prone as the number of steps grows. A scikit-learn pipeline chains the steps into a single estimator, so that fit fits every step on the data it is given and predict applies them all in order. Scikit-learn Basics explains pipelines in detail; the examples below use make_pipeline.

A baseline and a first model

The first model to evaluate is a baseline, a trivial solution that any useful model must beat. For a classification task, the simplest baseline predicts the most frequent class. The first real model here is logistic regression on standardised features.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import balanced_accuracy_score
from sklearn.pipeline import make_pipeline

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)

for name, est in [("baseline", baseline), ("logistic", model)]:
    score = balanced_accuracy_score(y_val, est.predict(X_val))
    print(name, round(score, 3))
# baseline 0.5
# logistic 0.991

print(round(baseline.score(X_val, y_val), 3))  # accuracy: 0.626

The baseline predicts "benign" for every sample. Its balanced accuracy is exactly 0.5, while its plain accuracy, returned by score, is 0.626. This is why the metric in the statement must be the one used on validation data: under plain accuracy, a model that has learned nothing already appears to score 63%. In an IOAI task, the statement's own baseline plays the same role, and scores are normalised against it, as described in Terminology. The models themselves are covered in Logistic Regression, K-Nearest Neighbors, Decision Trees and Random Forests.

Iterating on models

With a baseline and a validation set in place, candidate models can be compared on equal terms. Each candidate below is trained on the same training split and scored on the same validation split.

The first two candidates are logistic regression with two values of its hyperparameter C. C controls regularisation, a penalty that keeps the model's weights small so that it does not fit noise in the training data. Smaller values of C apply a stronger penalty and give a simpler model; the default is C=1.0. The remaining candidates are 5-nearest neighbours, a decision tree limited to depth 4, and a random forest with its default settings. Each model, and the role of its hyperparameters, is explained in its own module, starting with Logistic Regression; here only their validation scores are compared.

from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier

candidates = {
    "logistic, C=1": make_pipeline(
        StandardScaler(), LogisticRegression(C=1.0)),
    "logistic, C=0.1": make_pipeline(
        StandardScaler(), LogisticRegression(C=0.1)),
    "5-NN": make_pipeline(StandardScaler(), KNeighborsClassifier()),
    "tree, depth 4": DecisionTreeClassifier(
        max_depth=4, random_state=0),
    "random forest": RandomForestClassifier(random_state=0),
}
for name, est in candidates.items():
    est.fit(X_train, y_train)
    score = balanced_accuracy_score(y_val, est.predict(X_val))
    print(f"{name:17}{score:.3f}")

Small differences in such a table should be read with care. The validation set has 34 malignant and 57 benign samples, so a single misclassified malignant sample lowers the balanced accuracy by $1/(2 \times 34) \approx 0.015$, and a single benign one by $1/(2 \times 57) \approx 0.009$. The two logistic regression models each misclassify a single validation sample, a benign one for C=1 and a malignant one for C=0.1, and the gap between their scores comes entirely from which class that sample belongs to. A difference of that size is within the noise of one split; Cross-Validation gives a more reliable comparison by averaging over several splits, and Hyperparameter Tuning describes how to search settings such as C systematically.

Producing the submission

Once a model has been chosen, it is usually refitted on all labelled data. The validation split was needed to make the choice, but after the choice is made, the extra 91 samples improve the final model. The refitted model then predicts the test inputs, and the predictions are written in the required format.

final_model = make_pipeline(StandardScaler(), LogisticRegression())
final_model.fit(X, y)  # all 455 labelled rows

test_pred = final_model.predict(test[features])
submission = pd.DataFrame({"id": test["id"], "target": test_pred})
submission.to_csv("submission.csv", index=False)

The file should then be read back and checked against every requirement in the statement. A few assertions catch most format errors before a submission is wasted.

sub = pd.read_csv("submission.csv")

assert list(sub.columns) == ["id", "target"]  # exact header
assert len(sub) == len(test)                  # one row per test row
assert (sub["id"] == test["id"]).all()        # same order
assert sub["target"].dtype == "int64"         # integers, not 1.0
assert sub["target"].isin([0, 1]).all()       # allowed values only

with open("submission.csv") as f:
    print(f.read().splitlines()[:3])
# ['id,target', '502,0', '109,0']

Common format errors include:

• an extra index column: DataFrame.to_csv writes the row index as an unnamed first column unless index=False is passed;
• floats instead of integers: a float array is written as 1.0 and 0.0;
• a header that was not requested, or a missing one;
• the wrong class encoding: 0 and 1 where the statement asks for −1 and 1;
• reordered rows: predictions no longer aligned with the test file.

In this example, the organisers then score the file against the hidden labels.

# Organiser side: score the submission against the hidden labels
hidden = pd.read_csv("hidden_labels.csv")
test_score = balanced_accuracy_score(hidden["target"], sub["target"])
print(round(test_score, 3))  # 0.976

The test score of 0.976 is slightly below the validation score of 0.991. A small drop of this kind is normal: the model was selected because it did well on the validation set, so the validation score is mildly optimistic.

Other submission formats

IOAI statements use a variety of formats. The following code writes three of them from the same predictions: a CSV file with one label per line and no header, as in Antique Painting Authentication; a JSON list of integers, as in Robot Chasing; and a zip archive of files.

import json
import zipfile

# One label per line, no header, classes coded -1 and 1
labels = np.where(test_pred == 1, 1, -1)
pd.Series(labels).to_csv("submissionA.csv", index=False, header=False)

# A JSON list of integers, in the order of the test rows
with open("predictions.json", "w") as f:
    json.dump(test_pred.tolist(), f)

# Files packed into one zip archive
with zipfile.ZipFile("submission.zip", "w") as zf:
    zf.write("submissionA.csv")

The call to tolist() is required: json.dump cannot write a NumPy array, and it cannot write a list of NumPy integers either, so the array must be converted to Python integers first.

Reproducibility and time limits

A contest notebook is run again by the grading system, often on hidden data, so it must produce its submission reliably on its own.

• Run from top to bottom. Before submitting, restart the kernel and run every cell in order. A notebook that works only because of cells run earlier in a different order will fail when graded.
• Fix random seeds. Pass random_state to every splitter and estimator that uses randomness. Two RandomForestClassifier(random_state=0) models fitted on the same data are identical; without the seed, they differ from run to run.
• Measure the runtime. When a statement's time limit includes training at grading time, the full pipeline must finish well within it on the grading hardware.

import time

start = time.perf_counter()
final_model.fit(X, y)
test_pred = final_model.predict(test[features])
print(f"{time.perf_counter() - start:.3f} s")

This prints the elapsed time in seconds. For logistic regression on 455 samples it is a small fraction of a second; for larger models or datasets, measuring it early prevents a submission from failing on time.