Unsupervised

Unsupervised#

Unsupervised methods find structure in data without labels. Two families dominate the operator’s working set: clustering (group similar points) and dimensionality reduction (compress while preserving structure). Both are building blocks for everything downstream: feature engineering for supervised models, visualization, anomaly detection, retrieval.

Clustering#

Algorithm

Detail

k-means

Partition into k clusters by Euclidean distance to centroids. Fast, simple, requires k, sensitive to outliers, assumes spherical clusters.

k-medoids (PAM)

Like k-means but the cluster center is an actual point. Robust to outliers; slower.

Gaussian Mixture Model (GMM)

Soft clustering. Each point has a probability of membership in each cluster. Fits via EM; handles elliptical clusters.

DBSCAN

Density-based. No k needed; finds clusters of arbitrary shape; labels low-density points as noise. Parameters: epsilon (neighbourhood radius), minPts.

HDBSCAN

Hierarchical DBSCAN. Stable across density variations; the modern default when shapes are arbitrary.

Hierarchical clustering (agglomerative)

Builds a tree by repeatedly merging closest pairs. Linkage (single, complete, average, Ward) controls shape. Useful for dendrogram visualization.

Spectral clustering

Project to eigenvectors of the graph Laplacian; cluster in that space. Captures non-convex shapes; expensive on large data.

Affinity Propagation

Self-selecting exemplars; no k needed; expensive.

BIRCH

Memory-efficient for very large datasets; pre-clusters then refines.

Mean Shift

Mode-seeking, no k needed; slow on large data.

Pick k-means as the baseline; switch to HDBSCAN when shapes are non-convex or outliers must be flagged; use GMM when probabilities of membership matter.

Picking k#

Elbow plot of within-cluster sum of squares against k. The knee is where adding clusters stops buying real reduction in inertia; that’s the operator’s default starting k.

        xychart-beta
    title "Elbow plot: inertia vs k (illustrative)"
    x-axis [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    y-axis "inertia" 0 --> 1000
    line [950, 620, 380, 220, 180, 160, 150, 142, 138, 135]

  • Elbow plot, inertia (within-cluster sum of squares) vs k; pick the bend.

  • Silhouette score, in-cluster vs out-of-cluster distance. Higher is better.

  • Gap statistic, compares inertia against a null reference.

  • Domain knowledge, often the strongest signal.

Dimensionality reduction#

Algorithm

Detail

PCA (Principal Component Analysis)

Linear projection onto axes of maximum variance. The default starting point. Cheap; deterministic; explains how much variance each component captures.

Kernel PCA

PCA in a kernel-induced space; captures non-linear structure.

Truncated SVD

PCA for sparse matrices (text, recommender). Same maths, different implementation.

ICA (Independent Component Analysis)

Finds statistically independent sources. Used in signal separation (EEG, audio).

NMF (Non-negative Matrix Factorization)

Factorise into non-negative parts. The “topic model” of choice before LDA; interpretable additive components.

t-SNE

Non-linear, preserves local neighbourhoods; the default visualization for high-dimensional points in 2D. Don’t read absolute distances or global geometry from t-SNE plots.

UMAP

Faster, preserves more global structure than t-SNE. The modern default for visualization and as a feature transformation.

Autoencoder

Neural-network encoder + decoder; the bottleneck is the reduced representation. Strong on images / sequences.

LDA (Linear Discriminant Analysis)

Supervised: projects to maximize class separability. Useful as a classification preprocessing step.

PCA in operator practice:

from sklearn.decomposition import PCA

pca = PCA(n_components=0.95)              # keep enough comps for 95% variance
X_red = pca.fit_transform(X_scaled)
print(pca.n_components_, sum(pca.explained_variance_ratio_))

Matrix factorisation#

Beyond PCA / SVD lie the broader matrix-factorisation methods used in collaborative filtering, topic modeling, and embedding extraction.

  • SVD, factorise A = U Σ V^T. Building block for PCA, LSA, truncated-rank approximation.

  • ALS, alternating least squares; the standard recommendation algorithm at scale (Spotify, Netflix-era).

  • LDA (Latent Dirichlet Allocation, distinct from Linear Discriminant Analysis), probabilistic topic model.

Density estimation#

Method

Detail

Histogram

Bin and count. Quick, sensitive to bin width.

KDE (Kernel Density Estimation)

Smooth non-parametric estimate. Bandwidth is the knob.

GMM (as density model)

Mixture of Gaussians estimates a smooth multimodal density.

Association rules#

For categorical / transactional data (“customers who bought X also bought Y”).

  • Apriori, classical algorithm; slow at scale.

  • FP-Growth, faster modern variant; supported in Spark.

  • Eclat, depth-first set intersection.

Metrics: support (how often the pattern occurs), confidence (how often the rule holds given the antecedent), lift (how much the rule beats random).

Pitfalls#

  • Scaling: most distance-based methods care about feature scale. Standardize first.

  • Mixed types: clustering with mixed numeric + categorical needs a careful distance metric (Gower) or one-hot encoding first.

  • Curse of dimensionality: in high dimensions, distances concentrate; clustering loses meaning. Reduce first.

  • Reading t-SNE / UMAP plots: visual proximity is local; do not infer global cluster identity.

Implementations#

Tool

Detail

scikit-learn

All the above except HDBSCAN (separate package), UMAP (separate package), large-scale ALS.

hdbscan

Reference HDBSCAN implementation.

umap-learn

The Python UMAP package.

PyOD

Anomaly detection bundle; many of the same primitives appear there.

PySpark MLlib

Distributed k-means, GMM, ALS, LDA for warehouse-scale data.

References#

Contents