BM25: The Relevance Algorithm Behind Elasticsearch

BM25 is the default relevance algorithm used by Elasticsearch for most text fields. When a WooCommerce shopper searches for a phrase such as waterproof hiking boots, BM25 helps Elasticsearch decide which product documents should appear first by comparing the query terms with the content of each document.

For WooCommerce agencies, understanding BM25 is important because search quality is affected by more than whether a product contains a keyword. Term frequency, document length, term rarity, field configuration, analyzers, and query structure all influence the final _score.

What BM25 measures

BM25 evaluates three primary signals:

Term frequency: how often a searched term appears in a field.
Inverse document frequency: how rare the term is across the index.
Field-length normalization: whether the matching field is unusually short or long compared with other documents.

A simplified BM25 term score is:

score = IDF * (TF * (k1 + 1)) / (TF + k1 * (1 - b + b * fieldLength / averageFieldLength))

Elasticsearch uses Lucene’s BM25 implementation. Its inverse document frequency is commonly represented as:

IDF = log(1 + (documentCount - documentFrequency + 0.5) / (documentFrequency + 0.5))

The total score for a document is influenced by the matching terms and by the query type. A document matching several important query terms will generally score higher than one matching only a single term.

Term frequency is not a simple repetition counter

BM25 rewards a term appearing more than once, but it deliberately limits the benefit of repeated occurrences. This prevents a product description from ranking highly merely because the same word has been copied many times.

For example, suppose two products contain the word leather. One product mentions it once in the title and another repeats it ten times in a long description. The second product does not automatically win by a factor of ten. The k1 parameter controls how quickly the benefit of additional occurrences approaches its limit.

This is useful for product search because keyword stuffing in imported WooCommerce descriptions should not dominate relevance. However, repetition can still matter when the query matches several terms and the repeated terms occur in a field that has been given a strong boost.

Inverse document frequency makes rare terms valuable

A term found in almost every product is not very discriminating. Words such as product, new, or available may contribute little because they have a high document frequency. A term such as a specific model number or uncommon material may contribute much more.

For a WooCommerce catalogue, the difference is significant:

shoe may occur in thousands of products and provide a modest signal.
gore-tex may occur in far fewer products and provide a stronger signal.
A distinctive SKU or model identifier may be highly useful when it is indexed as searchable text or queried in an exact-match field.

Document frequency is calculated within the relevant Elasticsearch shard. As a result, scores can vary across shards, especially in small or unevenly distributed indexes. A well-sized production index and appropriate shard design reduce the practical impact, but _score should not be treated as a universal value across different indexes.

Why document length matters

Without length normalization, a long product description would have more opportunities to contain a query term than a short, focused product title. BM25 accounts for this by comparing the matching field length with the average length of that field.

The b parameter controls the strength of this normalization:

b = 0 disables length normalization.
b = 1 applies full normalization.
The default value, b = 0.75, applies a substantial but not complete correction.

Length normalization is applied per field, not across the entire product document. A long description does not directly make a short title less relevant when the title and description are queried as separate fields. This is one reason field-level boosts and a deliberate mapping are usually more useful than combining every product property into one large text field.

The k1 and b parameters

BM25 exposes two principal tuning parameters:

k1

k1 controls term-frequency saturation. A higher value allows repeated occurrences to contribute for longer before reaching saturation. A lower value makes the first occurrence more important and reduces the effect of repetition.

b

b controls document-length normalization. A higher value penalizes long fields more strongly when they contain the same matching terms as shorter fields. A lower value makes field length less important.

Elasticsearch’s default BM25 settings are generally a good starting point:

k1 = 1.2<br>b = 0.75

Changing these values should be based on measured search behavior rather than intuition. An agency might test a lower b value if long, editorially rich product descriptions are being unfairly suppressed, or test a lower k1 value if repeated marketing language is affecting results. Similarity changes require index configuration and typically require reindexing existing data.

Configuring BM25 in an index

BM25 is normally used automatically for analyzed text fields. A custom similarity can be defined when an index is created:

{<br>  "settings": {<br>    "similarity": {<br>      "woocommerce_bm25": {<br>        "type": "BM25",<br>        "k1": 1.2,<br>        "b": 0.75<br>      }<br>    }<br>  },<br>  "mappings": {<br>    "properties": {<br>      "title": {<br>        "type": "text",<br>        "similarity": "woocommerce_bm25"<br>      }<br>    }<br>  }<br>}

In most WooCommerce implementations, the default similarity is sufficient. Custom settings should be isolated to fields where the business case is clear. Do not use BM25 tuning to compensate for a poor analyzer, duplicated content, incorrect field mapping, or a query that gives too much weight to the wrong fields.

Field boosts are often more important than BM25 tuning

Search intent is frequently expressed in the product title, SKU, category, or attributes. A generic description should not have the same influence as a carefully maintained title. A multi_match query can express this priority:

{<br>  "query": {<br>    "multi_match": {<br>      "query": "waterproof hiking boots",<br>      "fields": [<br>        "name^4",<br>        "sku^5",<br>        "categories^2",<br>        "attributes^2",<br>        "description"<br>      ],<br>      "type": "best_fields"<br>    }<br>  }<br>}

The exact effect depends on the query type. With best_fields, the strongest matching field is emphasized, while most_fields can reward a document that matches across several analyzed fields. cross_fields is useful when terms may be distributed across fields that together represent one logical field, but it requires careful analyzer and field design.

Boosting should reflect the catalogue’s business meaning. If the SKU is mainly used for exact lookup, use a dedicated keyword field and an exact or prefix query alongside the BM25 text query. Do not expect BM25 alone to provide reliable exact-identifier behavior.

Analyzers determine what BM25 can match

BM25 scores analyzed terms, not raw strings. The analyzer determines whether a query can match a product at all and which terms contribute to the score.

Important analyzer decisions include:

Lowercasing so that Boots and boots produce compatible terms.
Tokenization of model numbers, hyphenated terms, and compound product names.
Stopword handling for terms such as the or for.
Stemming or lemmatization for related word forms.
Synonym handling for terms such as sneakers and trainers.

For WooCommerce catalogues, model numbers and technical attributes deserve special testing. A standard analyzer may split a value such as ABC-123 in a way that does not support the desired lookup behavior. A common mapping pattern is to keep an analyzed field for normal search and a keyword subfield for exact matching, for example sku and sku.keyword.

BM25 and WooCommerce product content

When indexing WooCommerce data, separate searchable concepts into fields instead of concatenating every value into one description. A practical product document may include:

name for the product title.
description and short_description for editorial content.
sku for product identifiers.
categories and tags for taxonomy terms.
attributes for values such as color, size, material, and compatibility.
Dedicated keyword fields for filters, facets, and exact values.

Use BM25 text queries for relevance and structured filters for catalogue constraints. For example, a color filter should normally be a term filter against a normalized keyword field, not a free-text BM25 query. Price ranges, stock status, visibility, product type, and taxonomy constraints should also be handled as filters where appropriate.

Inspecting why a product ranked

When a client reports that an unexpected product ranks first, the _score alone is not enough to diagnose the issue. Use Elasticsearch’s _explain API for a specific document and query:

GET products/_explain/product-123<br>{<br>  "query": {<br>    "multi_match": {<br>      "query": "waterproof hiking boots",<br>      "fields": ["name^4", "description"]<br>    }<br>  }<br>}

The explanation shows matching terms, inverse document frequency, term frequency, field-length normalization, and boosts. It can reveal issues such as a query matching only a generic description, an unexpectedly long field, an analyzer that removed an important term, or a title boost that is too weak.

The _termvectors API is also useful for checking which terms were indexed for an individual product. This is particularly valuable when debugging WooCommerce attributes, SKU tokenization, HTML-stripped descriptions, or synonym configuration.

Practical relevance testing

BM25 changes should be evaluated with representative search queries rather than a few manual examples. Build a test set containing:

Product-name searches.
Category and attribute searches.
SKU and model-number searches.
Synonyms and spelling variations.
Queries containing common words and rare technical terms.
Queries with competing products that differ in title and description quality.

Track metrics such as precision at the first result, recall for known products, click-through rate, add-to-cart rate, and conversion rate. Segment the results by query type because a change that improves descriptive searches may damage exact SKU lookups.

For agencies managing multiple stores, keep the similarity, analyzer, mapping, query, and ranking rules versioned together. A relevance change is not reproducible if only the query is stored while the index mapping and analyzer configuration remain undocumented.

Table of contents :