Supervised

Supervised#

Supervised learning fits a function f(x) -> y from labeled examples. Two problems:

  • Regression when y is continuous (price, latency, score).

  • Classification when y is a category (spam / not, benign / malicious, label A / B / C).

The operator’s working set covers a handful of methods that together solve most tabular and structured problems.

Linear models#

Method

Use

Linear regression (OLS)

Continuous y modeled as a linear combination of features. Interpretable; needs little data; baseline before anything fancier.

Ridge (L2) / Lasso (L1) / Elastic Net

Regularised linear regression. Ridge keeps all coefficients small; Lasso zeros some out (feature selection); Elastic Net interpolates.

Logistic regression

Binary classification through a logit link. The linear-model classifier counterpart. Outputs calibrated probabilities.

Multinomial / softmax regression

Multi-class generalisation.

Generalised linear models (GLM)

Family of which the above are members. Poisson regression for counts, gamma regression for positive-skewed continuous, etc.

Regularisation is the default; pure OLS is fine for the operator’s first pass on small, clean data.

Tree-based models#

Method

Use

Decision tree

Recursive split on a feature. Interpretable single-tree models; weak on their own.

Random forest

Bag of trees averaged. Robust, low tuning, decent default on tabular data.

Gradient-boosted trees

Trees added sequentially, each correcting the previous error. XGBoost, LightGBM, CatBoost. The current default winning model on tabular data.

Extra Trees

Like random forest with extra randomisation in splits. Fast.

Boosting (LightGBM in particular) is the operator’s tabular default in 2026. Trains fast, handles categoricals natively (LightGBM, CatBoost), tolerates missing values, ranks feature importance for free.

import lightgbm as lgb

train = lgb.Dataset(X_train, label=y_train)
valid = lgb.Dataset(X_val, label=y_val, reference=train)
params = dict(objective="binary", metric="auc", num_leaves=63, learning_rate=0.05)
model = lgb.train(params, train, num_boost_round=500,
                  valid_sets=[valid], callbacks=[lgb.early_stopping(50)])

Distance-based#

Method

Use

k-nearest neighbors (kNN)

Classify by majority vote among the k nearest training points. Strong baseline; slow at query time on large data; needs feature scaling.

Support vector machine (SVM)

Maximum-margin classifier; kernels for non-linear boundaries (RBF, polynomial). Strong on small / medium datasets; slow training at scale.

Probabilistic#

Method

Use

Naive Bayes

Assumes feature independence given the class. Famously bad assumption, famously useful in text classification. Cheap, well-calibrated on balanced data.

Gaussian process regression

Bayesian non-parametric regression with uncertainty estimates. Right when the operator needs a smooth predictor that says how confident it is.

Neural networks#

For tabular data, gradient-boosted trees usually beat neural networks. Neural networks are the operator’s default when the input is:

  • Images, CNNs (ResNet, EfficientNet) or vision transformers.

  • Sequences / text, transformers (BERT, RoBERTa) for classification; large language models for everything else (Text).

  • Audio, conv-based or transformer-based.

  • Graphs, GNNs (Graph).

For training, PyTorch is the default; transformers and timm are the high-level libraries for pretrained models. For fine-tuning rather than from-scratch training, see Agentic.

Pipelines#

A reusable supervised-learning recipe:

  1. Split train / validation / test. Time-respecting if the data is temporal.

  2. Feature engineer with the train set only. Imputation, scaling, encoding categoricals (one-hot, target, embedding). Save the transformer.

  3. Fit the model on train; early-stop on validation; evaluate once on test.

  4. Calibrate classifier probabilities (Platt scaling, isotonic regression) if downstream code uses them as probabilities.

  5. Inspect with feature importance and SHAP values; spot training-set / target-leakage bugs.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

pre = ColumnTransformer([
  ("num", StandardScaler(), num_cols),
  ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols),
])
pipe = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000))])
pipe.fit(X_train, y_train)
pipe.score(X_test, y_test)

Evaluation#

Metric

Use

MAE / RMSE / MAPE

Regression error. MAE robust, RMSE penalises large errors more, MAPE percent-based (and broken near zero).

R^2

Fraction of variance explained.

Accuracy

Classification, balanced classes only. Misleading otherwise.

Precision / recall / F1

Imbalanced classification. Precision = true / predicted-positive; recall = true / actual-positive.

ROC-AUC

Threshold-independent classifier ranking. Heavy in class-imbalanced settings; use PR-AUC instead.

PR-AUC

Precision-recall curve area. The right default for imbalanced binary problems.

Log loss / cross-entropy

Penalises confidently wrong predictions; the loss most classifiers actually optimize.

ROC curves for two candidate classifiers against the diagonal no-skill baseline. The operator wants the curve that bows hardest toward the top-left corner.

        xychart-beta
    title "ROC curves: two models vs baseline (illustrative)"
    x-axis [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
    y-axis "TPR" 0 --> 1
    line [0, 0.45, 0.66, 0.78, 0.85, 0.90, 0.93, 0.96, 0.98, 0.99, 1.0]
    line [0, 0.22, 0.40, 0.55, 0.66, 0.74, 0.81, 0.87, 0.92, 0.97, 1.0]
    line [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

Learning curves show training and validation loss as the training set grows. When the two converge, more data is not the lever; when they diverge, it is.

        xychart-beta
    title "Learning curves: training vs validation loss (illustrative)"
    x-axis [100, 500, 1000, 2500, 5000, 10000, 25000, 50000]
    y-axis "loss" 0 --> 1
    line [0.05, 0.12, 0.18, 0.22, 0.25, 0.27, 0.28, 0.29]
    line [0.85, 0.62, 0.50, 0.42, 0.37, 0.34, 0.32, 0.31]

Pitfalls#

  • Leakage: target information in features. Catches even experienced teams. Suspect any feature with suspiciously perfect importance.

  • Imbalanced classes: a 99%-accurate classifier on a 1%-positive problem is the trivial “always predict negative” baseline.

  • Concept drift: the data-generating process shifts over time. Plan for retraining cadence and drift detection.

  • Multicollinearity: highly correlated features destabilise linear coefficients but rarely the predictions themselves.

Implementations#

Tool

Detail

scikit-learn

The Swiss-army knife. Every method above except boosting (use the specialized libraries).

XGBoost / LightGBM / CatBoost

Production tabular boosting.

PyTorch

Default deep-learning framework.

PyTorch Lightning / Hugging Face Trainer

Training-loop boilerplate over PyTorch.

References#

Contents