4.2.1Scikit-learn Basics

Up to this point, the guide has described models and problems in words and mathematics. From here on, almost every model is trained in code, and nearly all of that code uses one library: scikit-learn. Linear regression, decision trees, k-means, random forests, cross-validation and most evaluation metrics are all a few imports away.

What makes scikit-learn easy to learn is that everything in it follows the same small set of rules. A model is created with its settings, trained with fit and asked for predictions with predict; a preprocessing step learns from the training data and is then applied unchanged to new data. Once these rules are familiar, a model you have never used before takes only minutes to try.

Learning the rules before the models means that the later modules can focus on how each model works rather than on how to call it. This module also covers the errors that beginners meet most often, which can otherwise cost a lot of time in a contest.

Scikit-learn is the standard Python library for classical machine learning. The IOAI 2026 Contest Rules name it, together with PyTorch, as one of the two core AI and machine learning libraries available in the contest environment.

This module covers the interface in practice. Terminology introduces the idea of an estimator and its two main methods, fit and predict, and explains the difference between scikit-learn's parameters and attributes. The models themselves are explained in the modules of section 4.3, starting with Linear Regression.

How the library is organised

Scikit-learn is divided into submodules by purpose, and classes are imported from the submodule that contains them, for example from sklearn.linear_model import LogisticRegression.

Other submodules used later in the guide include sklearn.cluster (KMeans), sklearn.decomposition (PCA), sklearn.svm (SVC), sklearn.naive_bayes (GaussianNB) and sklearn.feature_extraction.text (TfidfVectorizer).

Estimators

Fitting, predicting and scoring

An estimator, as defined in Terminology, is any scikit-learn object that learns from data through a fit method. Models such as logistic regression are estimators, but so are preprocessing tools such as StandardScaler.

A supervised model generally follows this pattern:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

The model class and its settings change from one algorithm to another, but the calls to fit and predict usually stay the same. We will see this recurring pattern throughout the classical machine learning model modules.

The examples in this section use the Iris dataset, split into 120 training and 30 test flowers.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=1
)

clf = LogisticRegression()           # hyperparameters are set here
print(clf.fit(X_train, y_train) is clf)  # True
print(clf.predict(X_test.iloc[:5]))  # [2 0 1 0 0]
print(clf.score(X_test, y_test))     # 0.9666666666666667

Five conventions are visible here.

• fit learns from data. A supervised estimator receives a two-dimensional feature matrix X_train and a label vector or matrix y_train. Unsupervised estimators and transformers may need only X_train.
• fit returns the estimator itself. This allows a model to be created and trained in one expression, such as LogisticRegression().fit(X_train, y_train).
• predict applies the fitted model to new samples. Here, clf.predict(X_test) produces one predicted class for each test sample.
• score returns a default metric. For classifiers such as LogisticRegression, it is accuracy, the fraction of correct predictions.
• Hyperparameters are set when the estimator is created. LogisticRegression() uses the defaults; LogisticRegression(C=0.1) would change one.

from sklearn.metrics import accuracy_score

y_pred = clf.predict(X_test)
print((y_pred == y_test).sum(), len(y_test))  # 29 30
print(accuracy_score(y_test, y_pred))         # 0.9666666666666667

For a regressor, score returns the coefficient of determination $R^2$ instead:

$$R^2 = 1 - \frac{\sum_{i} \left(y_i - \hat{y}_i\right)^2}{\sum_{i} \left(y_i - \bar{y}\right)^2},$$

where $\bar{y}$ is the mean of the true labels. The fraction compares the model's squared error with that of always predicting $\bar{y}$, so $R^2 = 1$ for perfect predictions and $R^2 = 0$ for a model no better than the mean. Model Evaluation Metrics explains this metric in more detail.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

Xd, yd = load_diabetes(return_X_y=True)
Xd_train, Xd_test, yd_train, yd_test = train_test_split(
    Xd, yd, test_size=0.2, random_state=0
)
reg = LinearRegression().fit(Xd_train, yd_train)

print(round(reg.score(Xd_test, yd_test), 3))              # 0.332
print(round(r2_score(yd_test, reg.predict(Xd_test)), 3))  # 0.332

mse_model = mean_squared_error(yd_test, reg.predict(Xd_test))
mse_mean = ((yd_test - yd_test.mean()) ** 2).mean()
print(round(mse_model, 1), round(mse_mean, 1))  # 3424.3 5127.9
print(round(1 - mse_model / mse_mean, 3))       # 0.332

A task is scored by the metric in its statement, which is often neither accuracy nor $R^2$. The functions in sklearn.metrics compute the common metrics directly from true and predicted labels. Model Evaluation Metrics explains when to use them, while the scikit-learn model evaluation guide documents their exact implementation.

Probabilities and decision scores

Many classifiers can report how confident each prediction is.

proba = clf.predict_proba(X_test.iloc[:3])
print(clf.classes_)          # [0 1 2]: the column order of proba
print(proba.round(3))
# [[0.    0.019 0.981]
#  [0.969 0.031 0.   ]
#  [0.292 0.707 0.001]]
print(proba.sum(axis=1))     # [1. 1. 1.]: each row sums to 1
print(proba.argmax(axis=1))  # [2 0 1]: the most probable class
print(clf.decision_function(X_test.iloc[:3]).round(2))
# [[-7.8   1.93  5.87]
#  [ 6.66  3.22 -9.89]
#  [ 1.56  2.44 -4.  ]]

• predict_proba returns one row per sample and one column per class, in the order given by the fitted attribute classes_. Its rows sum to 1. For this logistic regression model, predict returns the class with the highest probability. These values are the model's estimated probabilities for the possible classes. They are not guaranteed to match the true frequencies unless the model is well calibrated.
• decision_function returns raw scores before they are converted to probabilities. For a binary problem it returns one score per sample, with shape (n_samples,); for three classes, as here, one score per class.

Not every estimator provides both methods. In scikit-learn 1.9.1, LogisticRegression has both; KNeighborsClassifier, DecisionTreeClassifier and RandomForestClassifier have predict_proba but not decision_function; regressors such as LinearRegression have neither.

Fitted attributes

Everything an estimator learns during fit is stored in attributes whose names end in an underscore.

print(clf.n_features_in_)     # 4
print(clf.feature_names_in_)
# ['sepal length (cm)' 'sepal width (cm)' 'petal length (cm)'
#  'petal width (cm)']
print(clf.coef_.shape)        # (3, 4): one weight per class and feature

feature_names_in_ exists only when the estimator was fitted on a pandas DataFrame, and scikit-learn uses it to check that later inputs have the same columns. Calling predict on an estimator that has not been fitted raises an error:

from sklearn.exceptions import NotFittedError

try:
    LogisticRegression().predict(X_test)
except NotFittedError as err:
    print(err)
# This LogisticRegression instance is not fitted yet. Call 'fit' with
# appropriate arguments before using this estimator.

Transformers

Scaling

StandardScaler standardises each feature: it subtracts the feature's mean $\mu$ and divides by its standard deviation $\sigma$, both learned from the training data, so that a value $x$ becomes $z = (x - \mu)/\sigma$.

import numpy as np
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)  # learn mu and sigma, apply
X_test_s = scaler.transform(X_test)        # apply the same mu and sigma

print(scaler.mean_.round(2))   # [5.87 3.06 3.79 1.2 ]
print(scaler.scale_.round(2))  # [0.84 0.42 1.78 0.77]
print(np.allclose(X_train_s.mean(axis=0), 0))  # True
print(np.allclose(X_train_s.std(axis=0), 1))   # True
print(X_train_s[:5, 2].round(2))  # petal length of five flowers
# [-1.28 -1.34  0.73  0.34 -0.11]

The learned means and standard deviations are stored in mean_ and scale_, one per feature. For petal length they are 3.79 cm and 1.78 cm, so the first five training flowers change as follows:

A petal as long as the mean becomes 0, and one standard deviation longer becomes 1. After the transform, every column of X_train_s has mean 0 and standard deviation 1, as the two np.allclose checks confirm. The later pipeline examples show how to keep this scaling step together with the model.

Missing values

A dataset may contain many missing values. Data Cleaning covers how to find them and decide how they should be handled. Once you decide to replace them, scikit-learn provides a simple utility for doing so.

SimpleImputer replaces missing values, written as np.nan, with a statistic of each column learned from the training data.

from sklearn.impute import SimpleImputer

A_train = np.array([[1.0, 10.0],
                    [np.nan, 20.0],
                    [3.0, np.nan],
                    [5.0, 40.0]])
A_test = np.array([[np.nan, np.nan]])

imputer = SimpleImputer(strategy="median")
print(imputer.fit_transform(A_train))
# [[ 1. 10.]
#  [ 3. 20.]
#  [ 3. 20.]
#  [ 5. 40.]]
print(imputer.statistics_)        # [ 3. 20.]: training medians
print(imputer.transform(A_test))  # [[ 3. 20.]]

The test row is filled with the medians of the training columns, not with statistics of the test data.

Categorical features

Most estimators require numeric input, so categorical values must first be encoded as numbers. OneHotEncoder provides a clean way to turn one column of categories into one binary column per category. The training data below has four weather values; the test data has two, and one of them, fog, never appears in training.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

w_train = pd.DataFrame({"weather": ["clear", "rain", "clear", "snow"]})
w_test = pd.DataFrame({"weather": ["rain", "fog"]})

encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
train_encoded = encoder.fit_transform(w_train)  # learns the categories
test_encoded = encoder.transform(w_test)        # reuses them
print(encoder.get_feature_names_out())
# ['weather_clear' 'weather_rain' 'weather_snow']

Each row becomes three columns, one per category seen in training:

The categories are learned during fit. A category that appears only in the test data, here fog, is encoded as all zeros because of handle_unknown="ignore". With the default, handle_unknown="error", it raises ValueError: Found unknown categories ['fog'] in column 0 during transform. The option sparse_output=False returns an ordinary array instead of the default sparse matrix, which is easier to inspect.

Pipelines

A pipeline chains transformers and a final estimator into a single estimator. Calling fit on the pipeline fits each step in turn on the output of the previous one, and predict or score passes new data through the fitted steps. The rule of fitting preprocessing on training data only is then followed automatically.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline, make_pipeline

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("knn", KNeighborsClassifier()),
])
pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))  # 0.9666666666666667
print(pipe.named_steps["scaler"].mean_.round(2))
# [5.87 3.06 3.79 1.2 ]

Pipeline takes a list of (name, estimator) pairs, and named_steps gives access to each fitted step. The scaler inside the pipeline learned exactly the same means as the scaler fitted by hand on X_train. make_pipeline builds the same object without explicit names, naming each step after its class in lowercase.

Hyperparameters of a step are addressed as step__parameter, with two underscores. This is how pipelines are tuned.

auto = make_pipeline(StandardScaler(), KNeighborsClassifier())
print(list(auto.named_steps))
# ['standardscaler', 'kneighborsclassifier']

pipe.set_params(knn__n_neighbors=15)
print(pipe.get_params()["knn__n_neighbors"])                  # 15
print(auto.get_params()["kneighborsclassifier__n_neighbors"])  # 5

Columns of different types

Real tables mix numeric and categorical columns, often with missing values, and each kind needs different preprocessing. The six deliveries below, made up for this example, record a distance, a weight, a vehicle type and the weather, and whether each delivery was late. One weight and one weather value are missing.

In code, the table is a DataFrame with np.nan in place of each missing value, and the labels are a separate list:

deliveries = pd.DataFrame({
    "distance_km": [19.5, 8.8, 2.2, 1.5, 24.6, 12.0],
    "weight_kg": [4.4, np.nan, 2.3, 0.6, 6.8, 3.1],
    "vehicle": ["van", "van", "car", "car", "bike", "bike"],
    "weather": ["clear", "rain", "clear", np.nan, "clear", "snow"],
})
late = [1, 0, 0, 0, 1, 1]

The numeric columns need their missing values filled and their scale standardised; the categorical columns need their missing values filled and each category turned into its own column. A ColumnTransformer applies a separate transformer, or a small pipeline, to each group of columns and joins the results side by side:

from sklearn.compose import ColumnTransformer

preprocess = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"),
                          StandardScaler()),
     ["distance_km", "weight_kg"]),
    ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                          OneHotEncoder(handle_unknown="ignore",
                                        sparse_output=False)),
     ["vehicle", "weather"]),
])
table = preprocess.fit_transform(deliveries)
print(table.shape)  # (6, 8)

The result has one row per delivery and eight columns of numbers, shown here rounded to two decimals. preprocess.get_feature_names_out() names them num__distance_km, num__weight_kg, cat__vehicle_bike and so on; the table shortens those names.

• Numeric columns. The missing weight of delivery 1 was filled with the median of the known weights, 3.1 kg, so after standardising it equals the weight of delivery 5: both are −0.15.
• Categorical columns. The missing weather of delivery 3 was filled with the most frequent value, clear. Each vehicle type and each kind of weather then became its own column of 0s and 1s.
• Other columns. Columns not listed in any transformer are dropped, since the default is remainder="drop".

Joined with a model in one pipeline, the preprocessing is fitted together with the model, and new rows can be passed in their raw form, including missing values and a category that never appeared in training:

model = Pipeline([
    ("preprocess", preprocess),
    ("clf", LogisticRegression()),
])
model.fit(deliveries, late)

new = pd.DataFrame({
    "distance_km": [25.0, 3.0],
    "weight_kg": [np.nan, 2.5],
    "vehicle": ["scooter", "car"],  # "scooter" was not seen in fit
    "weather": ["snow", np.nan],
})
print(model.predict(new))                # [1 0]
print(model.predict_proba(new).round(2))
# [[0.17 0.83]
#  [0.79 0.21]]

The unseen vehicle scooter is encoded as 0 in all three vehicle columns, because of handle_unknown="ignore". Six deliveries are far too few to trust these probabilities; the point here is that the pipeline turns raw rows into predictions in one step.

Parameter names extend through every level of nesting:

params = model.get_params()
print(params["clf__C"])                                    # 1.0
print(params["preprocess__num__simpleimputer__strategy"])  # median

Cross-validation and parameter searches

sklearn.model_selection provides tools that fit a model many times on different parts of the data. cross_val_score splits the training data into $k$ parts, called folds, trains on $k - 1$ of them and scores on the remaining one, once for each fold. GridSearchCV repeats this for every combination of hyperparameter values in a grid. These tools give more reliable comparisons, but they require many model fits. Cross-Validation and Hyperparameter Tuning explain when that cost is useful. The example uses the scaled K-NN pipeline pipe and the Iris training data from above.

from sklearn.model_selection import GridSearchCV, cross_val_score

scores = cross_val_score(pipe, X_train, y_train, cv=5)
print(scores.round(3))          # [0.917 1.    1.    1.    0.958]
print(round(scores.mean(), 3))  # 0.975

param_grid = {
    "knn__n_neighbors": [1, 5, 15, 45],
    "knn__weights": ["uniform", "distance"],
}
search = GridSearchCV(pipe, param_grid, cv=5)
search.fit(X_train, y_train)
print(search.best_params_)
# {'knn__n_neighbors': 15, 'knn__weights': 'uniform'}
print(round(search.best_score_, 3))            # 0.975
print(round(search.score(X_test, y_test), 3))  # 0.967

The grid has $4 \times 2 = 8$ combinations, each evaluated with 5-fold cross-validation. best_score_ is the mean cross-validated accuracy of the best combination. After the search, the best combination is refitted on all of X_train, so search can be used directly to predict or score.

Reproducibility

Estimators and functions that involve randomness accept a random_state argument. With the same integer seed, the results are identical from run to run; with a different seed, or with the default None, they can differ.

from sklearn.ensemble import RandomForestClassifier

def forest_proba(seed):
    rf = RandomForestClassifier(n_estimators=50, random_state=seed)
    return rf.fit(X_train, y_train).predict_proba(X_test)

print((forest_proba(42) == forest_proba(42)).all())  # True
print((forest_proba(42) == forest_proba(7)).all())   # False

Reading the API reference

Each class in the API Reference has the same structure: a Parameters section listing the constructor arguments with their defaults, an Attributes section listing the values learned during fit, and a Methods section. The installed version and an estimator's current settings can also be checked in code.

import sklearn

print(sklearn.__version__)  # 1.9.1
print(KNeighborsClassifier().get_params())
# {'algorithm': 'auto', 'leaf_size': 30, 'metric': 'minkowski',
#  'metric_params': None, 'n_jobs': None, 'n_neighbors': 5, 'p': 2,
#  'weights': 'uniform'}

Defaults sometimes change between releases, and the IOAI contest environment uses pinned library versions. The documentation consulted should therefore match the version reported by sklearn.__version__; the version selector at the top of the documentation site switches between releases.

Common errors

Scikit-learn's error messages usually state the cause precisely. The table below lists the most frequent ones, with the key part of each message as printed by scikit-learn 1.9.1.

The first two errors arise from passing a single sample incorrectly:

row = X_test.iloc[0]  # one sample as a Series: a 1-D object
try:
    clf.predict(row)
except ValueError as err:
    print(err)
# Expected a 2-dimensional container but got <class 'pandas.Series'>
# instead. Pass a DataFrame containing a single row (i.e. single
# sample) or a single column (i.e. single feature) instead.

print(clf.predict(X_test.iloc[[0]]))  # a one-row DataFrame: [2]

Missing values are rejected by most estimators, but a pipeline with an imputer accepts them:

X_nan = X_train.copy()
X_nan.iloc[0, 0] = np.nan
try:
    LogisticRegression().fit(X_nan, y_train)
except ValueError as err:
    print(str(err).splitlines()[0])  # Input X contains NaN.

fixed = make_pipeline(SimpleImputer(), StandardScaler(),
                      LogisticRegression())
print(fixed.fit(X_nan, y_train).score(X_test, y_test))
# 0.9666666666666667

Some estimators accept missing values without imputation. In scikit-learn 1.9.1 these include DecisionTreeClassifier, RandomForestClassifier and HistGradientBoostingClassifier; the User Guide page on imputation lists them all.

Refitting a transformer on test data can change the number of features, because the test data may contain fewer categories:

late_train = [0, 1, 0, 1]
enc = OneHotEncoder(sparse_output=False).fit(w_train)  # 3 columns
lr = LogisticRegression().fit(enc.transform(w_train), late_train)

enc_test = OneHotEncoder(sparse_output=False).fit(w_test)  # wrong
try:
    lr.predict(enc_test.transform(w_test))
except ValueError as err:
    print(err)
# X has 2 features, but LogisticRegression is expecting 3 features
# as input.

Where scikit-learn fits

Most scikit-learn estimators run on the CPU and expect the whole dataset in memory as a feature matrix. The library is therefore the natural choice for tabular data and for models trained on features that have already been extracted. Raw images, audio and long texts are usually handled with neural networks in PyTorch, covered in the later parts of this guide, starting with PyTorch Basics. The two often work together: in the on-site Help BOBAI task of IOAI 2024, contestants received text encodings produced by a pretrained network, and the official solution classifies them with KNeighborsClassifier(n_neighbors=3, weights='distance').