Search

Search and similarity primitives answer “find items like X.” The operator runs them as building blocks in retrieval-augmented generation, recommendation, deduplication, link analysis, and classical full-text search. Two layers: keyword / lexical search on tokens, and vector / semantic search on embeddings. Modern systems use both.

Lexical search

Keyword scoring over a tokenised corpus.

Model	Detail
Boolean / inverted index	The base case. term => doc list; query is a set operation.
TF-IDF	Term frequency × inverse document frequency. Decades-old default; still strong for short queries on small corpora.
BM25	Modernized TF-IDF with document-length normalization and saturation. The reference lexical scoring function; what Elasticsearch / OpenSearch / Lucene use by default.
Language model retrieval (Dirichlet smoothing)	Probabilistic alternative to BM25. Competitive; less common in production.

BM25 score for query q over document d:

\[\text{BM25}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{\text{tf}(t, d) (k_1 + 1)} {\text{tf}(t, d) + k_1 \left(1 - b + b \cdot |d| / \text{avgdl}\right)}\]

Default parameters: k1 = 1.2, b = 0.75.

Vector search

Embed documents and queries into a shared vector space; rank by distance. The operator’s default for semantic similarity, deduplication, RAG retrieval, and “find me ones like this.”

Distance	Detail
Cosine similarity	a · b / (||a|| ||b||). Range [-1, 1]. Default for normalized text and image embeddings.
Dot product	For unit-normalized vectors, identical to cosine; cheaper.
Euclidean (L2)	Useful for non-normalized embeddings and visual / signal data.
Manhattan (L1)	More robust to outliers in dimensions.
Hamming	Bitwise; for binary hashes (LSH outputs).

Nearest-neighbor search

Exact and approximate algorithms for “give me the k closest vectors to q.”

Algorithm	Detail
Brute force (flat)	Exact; O(N) per query. Right up to about a million vectors per node.
kd-tree / ball tree	Exact, sub-linear in low dimensions. Falls back to brute force past ~20 dimensions.
LSH (Locality-Sensitive Hashing)	Hash similar items to the same bucket. Sub-linear approximate. The original scalable approach.
IVF (Inverted File Index)	Cluster the corpus; search only the closest few clusters at query time. Used by FAISS.
PQ (Product Quantization)	Compress vectors to short codes; search in the compressed space. Combined with IVF (IVF-PQ) for billion-scale.
HNSW (Hierarchical Navigable Small World)	Multi-layer proximity graph. Current SOTA on recall/throughput trade-off for high-dimensional vectors.
ScaNN	Google’s approach. Strong on large corpora.
DiskANN / Vamana	SSD-backed ANN; serves billion-scale on a single host.

A worked similarity query:

import numpy as np

def topk_cosine(query, corpus, k=10):
    q = query / np.linalg.norm(query)
    C = corpus / np.linalg.norm(corpus, axis=1, keepdims=True)
    scores = C @ q                            # cosine = dot product when normalized
    idx = np.argpartition(-scores, k)[:k]
    return idx[np.argsort(-scores[idx])]

For production, switch to FAISS / hnswlib / Annoy at the first sign of corpus size.

Hybrid search

Lexical and vector scores combined. The standard pattern in 2026:

1. Retrieve top-N from BM25.

2. Retrieve top-N from a vector index.

3. Merge with reciprocal rank fusion (RRF) or a learned reranker.

RRF(d) = sum over retrievers of 1 / (k + rank_r(d))

A cross-encoder reranker (a transformer that scores a query-document pair) on the top-K of the fused list usually gives the biggest precision lift; it’s slower but only runs on the candidates.

Recommendation

Approach	Detail
Content-based	Match items to user profile in some feature space. No cold-start problem on items.
Collaborative filtering (user-based, item-based)	“Users who liked X also liked Y.” Memory-based.
Matrix factorisation (ALS)	Learn user and item embeddings; dot product is the predicted rating. Production CF at scale.
Two-tower neural recommendation	Train a user tower and an item tower into a shared embedding space. Retrieve via ANN.
Graph-based (PinSage, GraphSAGE)	Use the user-item graph plus side features.
Sequential models (RNN, transformer)	Predict the next item from session history.

Search engines

Engine	Detail
Elasticsearch / OpenSearch	Lucene-based, mature, BM25 default, dense-vector support added 2023+.
Apache Solr	Lucene’s older sibling. Strong on classical IR; less momentum in 2026.
Vespa	Yahoo’s open-source engine. Native dense + sparse + structured retrieval; ranking with custom models.
Typesense / Meilisearch	Lightweight modern search; great defaults, smaller ops surface than Elastic.
FAISS / Annoy / hnswlib / ScaNN	Library-level vector indexes. The operator embeds these in their own service.
Qdrant / Weaviate / Milvus / Pinecone	Vector databases. Add metadata filters, durable storage, REST APIs.
pgvector / sqlite-vec	Vector support inside Postgres / SQLite. Right when the corpus fits and the operator wants one database.

Pitfalls

• Out-of-vocabulary in lexical search. Stem, lowercase, and normalize before indexing.

• Embedding drift when the embedding model changes. Either reindex or keep both versions during a migration.

• Recall vs latency on ANN. Every parameter (efSearch, nprobe, M) trades them. Benchmark on the operator’s own corpus, not the algorithm vendor’s.

• Metadata filtering post-hoc kills recall; do it inside the index (pre-filter) if the engine supports it.

• Cold start on recommendation. Content-based or popularity fallbacks until enough interactions accumulate.

References

• Text for the embedding side of vector models.

• Unsupervised for the matrix-factorisation substrate.

• RAG for RAG specifically.

• BM25 reference

• Introduction to Information Retrieval (Manning, Raghavan, Schütze)

• FAISS

• HNSW paper (Malkov & Yashunin)