Text

Text#

Text analytics turns unstructured language into features the operator can compute on. The catalog runs from cheap classical methods (regex, n-grams, TF-IDF) through embedding-based search to large language models. Each layer is a building block; modern systems compose all of them in a single pipeline.

For the LLM and agent layer specifically, see Agentic and LLM-Assisted Analysis. This page covers the analytical primitives.

Tokenisation#

Splitting raw text into countable units.

Method

Detail

Whitespace / regex

Cheap; loses punctuation handling and language-specific rules.

Word tokeniser

NLTK, spaCy. Handles contractions, punctuation, language-specific quirks.

Sentence segmentation

Splits a document at sentence boundaries.

Subword (BPE, WordPiece, SentencePiece)

The modern default; what tokenisers in transformers use. Fixed vocabulary, no OOV tokens.

Character n-grams

Robust to misspellings and morphology; right for short texts and noisy data.

Normalization#

  • Lowercasing, the default; preserve case when the model is case-sensitive (NER often is).

  • Stemming (Porter, Snowball), reduce inflected forms to a stem. Crude.

  • Lemmatisation, reduce to dictionary form using POS context. Accurate; slower.

  • Stopword removal, drop high-frequency function words; useful for classical models, harmful for modern embeddings.

  • Unicode normalization (NFC), prevents same-glyph-different- codepoints surprises.

Vector space models#

Represent documents and queries as vectors so the operator can compute distances and similarities. The bedrock of classical IR and the foundation modern embeddings sit on.

Model

Detail

Bag of words (BoW)

Vector of term counts. Ignores order. Dimension equals vocabulary size.

TF-IDF

Term frequency × inverse document frequency. Down-weights common terms; the operator’s classical default.

Hashed (HashingVectorizer)

Hash tokens to a fixed dimension; no vocabulary to maintain. Right for streaming and very large vocabularies.

LSA / LSI

SVD of the TF-IDF matrix. Reduces to a “topic” space that captures synonymy.

LDA (Latent Dirichlet Allocation)

Probabilistic topic model. Each document is a mixture of topics, each topic is a distribution over words.

Word2vec / GloVe / FastText

Dense word embeddings. Capture semantic similarity. FastText handles OOV via subwords.

Sentence / document embeddings

Sentence-BERT, OpenAI / Cohere / Voyage embeddings. The current default for similarity search and clustering.

Cosine similarity is the standard distance in vector-space models. For unit-normalized vectors it equals the dot product.

import numpy as np

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

For retrieval at scale, vectorise the corpus once and use an approximate-nearest-neighbor index (see Search).

Classification#

Pipeline: tokenise → normalize → vectorise → classify.

Pipeline

Use

TF-IDF + linear SVM / logistic regression

The pre-deep-learning default. Strong baseline; cheap; interpretable.

TF-IDF + gradient boosting

Often comparable to linear models on text; less interpretable.

FastText

Trains a shallow embedding + linear classifier end-to-end. Fast.

BERT fine-tune

Modern default. Pre-trained transformer, fine-tuned on labeled task. State-of-the-art with modest data.

LLM with few-shot prompts

Zero training. Right when labels are scarce or class set is fluid.

Sequence labeling#

Task

Detail

POS tagging

Mark each token with its part of speech.

Chunking / shallow parsing

Group tokens into non-overlapping phrases.

Named entity recognition (NER)

Mark spans (person, org, location, date). spaCy, Stanza, transformer-based.

Span extraction

Pull spans matching a question. Used in QA systems.

Models, CRF, BiLSTM-CRF, BERT-token-classification.

Information extraction#

Turn unstructured text into structured records.

  • Regex / patterns, the cheap option. Strong on highly structured text (logs, EDI, structured forms).

  • Rule-based with spaCy Matcher / displaCy, declarative patterns over token attributes.

  • OpenIE, off-the-shelf triple extraction.

  • LLM extraction, prompt-driven, schema-validated output. The right default in 2026 for low-volume / high-variability text.

Topic and theme analysis#

  • LDA, classical probabilistic topic model.

  • BERTopic, embedding-cluster-c-TF-IDF pipeline. Easier to interpret than LDA on modern text.

  • Top2Vec, embedding-cluster with HDBSCAN.

Summarisation, translation, generation#

Pre-2022 work used encoder-decoder transformers (BART, T5) for each task. In 2026 the operator’s default is a general-purpose LLM behind one of the providers in SDKs. The classical methods are still right for offline / on-prem / cost-sensitive deployments.

Pitfalls#

  • Encoding, always normalize to UTF-8 NFC. Mixed encodings break tokenisation silently.

  • Language drift, models trained on news fail on social media. Domain-adapt by fine-tuning or selecting an in-domain pretrained model.

  • Bias, every step (vocabulary, embeddings, training set) inherits and amplifies its training-data biases. Test on a fairness-relevant slice.

  • Sample size, a TF-IDF + logistic regression on 50k labeled examples often matches a fine-tuned BERT on 5k. Match method to data scale.

Implementations#

Tool

Detail

spaCy

Production NLP pipeline (tokenise, POS, NER, parse). Fast and stable.

NLTK

Reference library for classical NLP; good for teaching, slow at scale.

Stanza / CoreNLP

Stanford’s pipeline; strong on linguistic accuracy.

Hugging Face transformers / sentence-transformers

Pretrained transformer models for embeddings, classification, generation.

Gensim

Word2vec, FastText, LDA, similarity over large corpora.

scikit-learn

TF-IDF, HashingVectorizer, classifiers, dimensionality reduction.

References#

Contents