Statistics

Statistics#

Statistics is the substrate. Almost every later technique reduces to one of: estimate a parameter, test a hypothesis, fit a model, or quantify uncertainty. The operator’s working set is small and durable.

Descriptive statistics#

Summaries of one variable.

Measure

Use

Mean

The expected value when the distribution is roughly symmetric. Sensitive to outliers.

Median

The 50th percentile. Robust to outliers; default summary for skewed data.

Mode

Most frequent value. Useful for categorical data.

Variance / standard deviation

Spread. Standard deviation in the same units as the data.

Inter-quartile range (IQR)

Spread of the middle 50%. Robust.

Skewness, kurtosis

Shape: tail asymmetry, tail heaviness.

Percentiles (p50, p95, p99, p99.9)

The default summary for latency and any heavy-tailed metric.

Pair with histograms and box plots before reaching for averages alone; a single mean over a bimodal distribution lies.

Two variables#

Measure

Use

Covariance

Sign and rough strength of co-movement. Scale-dependent.

Pearson correlation

Linear association. -1 to +1. Sensitive to outliers and non-linearity.

Spearman correlation

Rank correlation. Captures monotone non-linear relationships.

Mutual information

Any (non-linear) statistical dependence. Non-parametric.

Distributions#

The distributions worth recognizing on sight.

Distribution

Where it shows up

Normal

Sums of many independent effects (central limit theorem). Heights, measurement error.

Log-normal

Multiplicative effects, file sizes, income, latency.

Exponential

Time between independent events (Poisson process). Inter-arrival.

Poisson

Counts of rare independent events in a window.

Binomial

Successes in N independent trials.

Power-law / Pareto

Heavy-tail counts: city sizes, file downloads, request rates.

Uniform

Equal probability across a range. Random sampling.

Inferential statistics#

Quantifying uncertainty about a population from a sample.

  • Standard error, the standard deviation of the sample statistic. The smaller, the more precise the estimate.

  • Confidence interval, the operator’s default uncertainty surface. A 95% CI is the range that traps the true value in 95% of resamples.

  • Bootstrap, resample with replacement; compute the statistic on each resample; the spread is the bootstrap CI. Works for any statistic the operator can compute. The right default when the theoretical sampling distribution is unknown.

  • p-value, the probability of seeing a statistic at least as extreme as the observed one if the null hypothesis is true. Not the probability the null is true. Read carefully.

  • Effect size, Cohen’s d or similar. Operator practice: report effect size and CI; let p-values play a supporting role.

Hypothesis tests#

Test

Use

t-test

Difference of means between two samples (or one sample vs a hypothesised mean).

Mann-Whitney U

Non-parametric two-sample shift test. No normality assumption.

Chi-squared

Independence in a contingency table; goodness-of-fit to a distribution.

Kolmogorov-Smirnov

Two distributions differ. Non-parametric.

ANOVA

Differences in means across many groups.

Permutation test

Generic shuffle-based test. Use when assumptions of the canned tests fail.

A/B testing#

Operator-grade A/B uses a fixed sample size computed in advance, randomised assignment, primary metric defined before the experiment, and a stop rule that does not peek. Sequential testing (always valid p-values, mSPRT) lets the operator monitor without inflating false-positive rate.

Bayesian view#

Bayes is the alternative to the frequentist machinery above. Prior + likelihood => posterior. Useful when:

  • The operator has informative prior knowledge.

  • Decision-making requires probability statements about the parameter (“there’s a 90% chance the new model is better”).

  • Hierarchical structure makes pooled estimates the natural fit.

Standard tools: PyMC, Stan, NumPyro.

Implementations#

Tool

Use

NumPy / SciPy / pandas

Python’s substrate. scipy.stats has every test above.

statsmodels

Regressions, hypothesis tests, time-series, with statistical detail R-style.

R

The reference for statistical computing.

DuckDB / SQL

percentile_cont, stddev_samp, corr, regr_*. The right place for descriptive stats over warehouse-scale tables.

Pitfalls#

  • Confusing correlation with causation. The single most durable statistics lesson.

  • Multiple comparisons. Test 20 hypotheses at p < 0.05 and one false positive is expected. Apply Bonferroni, Holm, or Benjamini-Hochberg corrections.

  • Survivorship bias. The sample the operator can measure is not always the sample they want to measure.

  • Simpson’s paradox. Aggregate trend can reverse the subgroup trend. Stratify before pooling.

References#

Contents