Search

Search engines build inverted indexes (mapping terms to the documents that contain them) to answer “which documents match this query?” fast. Modern search engines also handle vectors, faceting, ranking, and analytics on top of those indexes.

Inverted Index 101

A search engine pre-computes which terms appear in which documents.

"quick" -> [doc1, doc4, doc9]
"brown" -> [doc1, doc7]
"fox"   -> [doc1, doc4, doc12]

A query for “quick brown fox” intersects the postings lists and ranks the results by relevance.

Engines

The major search engines an operator may meet. Elasticsearch leads; OpenSearch is the AWS fork; Solr is the Lucene-based elder; Meilisearch and Typesense target small-to-medium scale with simpler ops; Quickwit and Vespa specialize in logs and ML-serving respectively.

• Elasticsearch, the long-time market leader.

• OpenSearch, AWS’s Elasticsearch fork after the license change.

• Meilisearch, developer-friendly, small-to-medium scale.

• Typesense, alternative; minimal config.

• Solr, the original Lucene-based engine.

• MeiliSearch, real-time, typo-tolerant.

• Quickwit, log-and-search; cloud-native.

• Vespa, search + ML serving.

Postgres / SQL Server / MySQL also have built-in full-text search; often “good enough” before adding a dedicated engine.

Relevance

The factors that decide which results rank first. TF-IDF and BM25 are the math; field weights, recency boosts, personalization, and business signals are the levers. Tuning relevance is usually the longest project on a search system; budget for it from day one.

Default ranking blends.

• Term frequency / inverse document frequency (TF-IDF).

• BM25, TF-IDF refined; the modern baseline.

• Field weights, title matches > body matches.

• Boosting, recency, popularity, business signals.

• Personalization, per-user preferences, click history.

Ranking is usually the longest-running tuning project on a search system; budget for it.

Indexing

The other half of search: getting documents into the index correctly. Analyzers tokenize and stem; per-field configuration tunes for titles versus bodies; multi-field indexing keeps multiple analyses of the same data. Pipelines range from synchronous writes to CDC to periodic rebuilds.

What goes into the index.

• Analyzers tokenize, lowercase, stem, remove stopwords, expand synonyms.

• Per-field configuration, different analyzers for titles vs. bodies, languages, codes.

• Multi-field indexing, same field, multiple analyses (raw + stemmed + ngrammed).

Indexing pipelines.

• Sync, application writes index alongside primary store. Simple but partial-failure-prone.

• Async via change-data-capture, Debezium / Kafka Connect pull changes from the primary database into the search index.

• Reindex from source, periodically rebuild from the source of truth.

Vector Search

Modern search engines now also store embedding vectors and answer approximate nearest-neighbor queries. Elasticsearch, OpenSearch, and Vespa add vector fields alongside the inverted index; Qdrant, Weaviate, Milvus, and Pinecone start vector-first; pgvector covers Postgres. Hybrid usually wins.

• Elasticsearch, dense_vector fields, HNSW indexes.

• OpenSearch, k-NN plugin.

• Vespa, first-class hybrid search.

• Qdrant, Weaviate, Milvus, Pinecone vector-first.

• pgvector, Postgres extension.

Hybrid search (lexical + semantic, then re-ranked) typically beats either alone.

Faceting and Aggregations

Search engines aggregate as a side product of querying.

• “How many results in each category?” (terms aggregation)

• “Distribution of price?” (histogram / range aggregation)

• “Trend over the last month?” (date histogram)

This is what makes them double as analytical tools for some workloads.

Operations

Running a search cluster has its own playbook. Sharding decisions are sticky; replication serves both availability and read throughput; heap sizing leaves room for the OS page cache; index lifecycle policies move data through tiers; and reindexing is a fact of life that schemas must accommodate.

Search clusters have their own operational profile.

• Sharding, choose carefully; reshaping later is expensive.

• Replication, for both availability and read throughput.

• Heap sizing, Lucene memory-maps a lot; leave RAM for the OS page cache.

• Index lifecycle, hot / warm / cold tiers, retention, rollover.

• Snapshots to object storage for backup.

• Reindexing is a fact of life; design schemas to support it.

Logs and Observability

Search engines double as log-analytics back ends in many shops. The classic ELK stack remains common; Loki offers a cheaper alternative with different log handling; OpenSearch mirrors Elastic; Quickwit specializes in cheap log search. All grow expensive once retention extends; budget index size.

Search engines double as the storage tier for log analytics in many shops.

• The classic ELK stack: Elasticsearch + Logstash + Kibana.

• Loki + Grafana, cheaper alternative; treats logs differently.

• OpenSearch + OpenSearch Dashboards.

• Quickwit + Grafana, log-focused.

Grow logs into search and they stay; budget index size accordingly.

Choosing

The decision tree for picking a search engine. Start with the database’s full-text features when LIKE runs out; move to Elasticsearch or OpenSearch when relevance and scale demand it; choose Meilisearch or Typesense for simpler ops; pgvector for entry-level vector workloads.

• For most apps that need richer search than LIKE: start with the primary database’s full-text features.

• When relevance, faceting, or scale outgrows that: Elasticsearch or OpenSearch.

• Smaller, simpler need: Meilisearch or Typesense.

• Vector / RAG: pgvector if you already have Postgres; a vector-first DB for very large vector workloads.

• Logs at scale: ELK / Loki / Quickwit / Grafana.