Table of contents :

Ranking Products with BM25

100-days-of-vespa-ai-woocommerce-028

Table of contents :

For WooCommerce product search, BM25 is a strong lexical baseline when customers search with meaningful words such as waterproof hiking jacket, USB-C dock, or organic cotton sheets. Vespa provides BM25 as a rank feature, allowing an agency to combine text relevance with product availability, price, stock status, sales performance, and other business signals.

What BM25 contributes to product search

BM25 scores how well a query matches a document. It considers term frequency, the rarity of each term across the index, and the length of the matched field. A rare term such as a specific model number can contribute more than a common term such as product. BM25 also limits the benefit of repeating the same term, which makes it more robust than a simple raw term-frequency score.

BM25 is a relevance model, not a merchandising policy. It does not know whether a product is in stock, profitable, on sale, or preferred by the store owner. Those concerns should be represented by separate rank features and combined deliberately with the lexical score.

Enable BM25 for product fields

BM25 must be enabled on the indexed string fields that will participate in ranking. A product document might contain separate fields for the name, short description, long description, brand, and searchable attributes.

schema product_search {<br>    document product_search {<br>        field product_id type string {<br>            indexing: attribute | summary<br>        }<br><br>        field name type string {<br>            indexing: index | summary<br>            index: enable-bm25<br>        }<br><br>        field description type string {<br>            indexing: index | summary<br>            index: enable-bm25<br>        }<br><br>        field brand type string {<br>            indexing: index | summary<br>            index: enable-bm25<br>        }<br><br>        field in_stock type bool {<br>            indexing: attribute | summary<br>        }<br>    }<br><br>    rank-profile lexical_bm25 {<br>        first-phase {<br>            expression: bm25(name) + 0.35 * bm25(brand) + 0.20 * bm25(description)<br>        }<br>    }<br>}

The exact field names and weights should reflect the catalog. Product names usually deserve the highest weight because they are concise and commercially meaningful. Descriptions are useful for recall but can contain boilerplate, so they commonly receive a lower weight.

Use a BM25 rank profile in a query

After deploying the schema, select the rank profile when querying Vespa. A typical search request uses a user query, a YQL expression that searches the relevant fields, and the BM25 rank profile.

GET /search/?yql=select * from sources product_search where userInput(@query)&query=waterproof hiking jacket&ranking=lexical_bm25&hits=24

For production integrations, construct the request with a proper HTTP client rather than concatenating unescaped query-string values. The search expression can also include structured filters for WooCommerce data.

select * from sources product_search where<br>    userInput(@query)<br>    and in_stock contains true<br>    and category_id contains 42

Filtering should remove products that cannot be shown or purchased. It should not be simulated by assigning a large arbitrary relevance value to an out-of-stock product. Keeping eligibility rules in the query makes the ranking behavior easier to test and explain.

Map WooCommerce fields to search fields

A WooCommerce indexing pipeline should normalize the fields before sending documents to Vespa. Useful inputs include the product name, SKU, brand, categories, attributes, variation labels, short description, and selected long-description text.

  • Product name: Index the customer-facing name and preserve important model terms.
  • SKU and model number: Keep an exact-match path for identifiers, because BM25 alone is not always the best behavior for short codes.
  • Brand: Index it separately so brand matches can receive a controlled weight.
  • Attributes: Flatten values such as color, material, size, and compatibility into searchable text, while also retaining structured fields for filters.
  • Descriptions: Remove repeated template content where possible and avoid indexing administrative or hidden text.
  • Categories and tags: Index them separately or as a carefully constructed field so category terms do not overwhelm the product name.

For variable products, decide whether the searchable document represents the parent product, each variation, or both. If customers search for a variation-specific value such as 128 GB or navy, that value must be present in the document being ranked. Inventory and purchasability can still be maintained as attributes used by filters.

Combine BM25 with exact matching

Catalog searches often contain identifiers that should be treated differently from ordinary prose. A query for ABC-123 should strongly favor a product whose SKU or model field is exactly ABC-123, even if another product contains the same tokens in a longer description.

One approach is to create an exact-match query term or a separate rank feature for normalized identifiers, then combine it with BM25 in the rank profile. The implementation depends on how the ingestion and query parsing layers represent identifiers, but the ranking principle is consistent: use BM25 for general lexical relevance and a high-confidence exact signal for SKU and model matches.

rank-profile product_relevance {<br>    first-phase {<br>        expression: 1.0 * bm25(name)<br>                  + 0.35 * bm25(brand)<br>                  + 0.20 * bm25(description)<br>                  + 5.0 * attribute(exact_identifier_match)<br>    }<br>}

The value of an exact-match signal should be validated against real queries. An excessively large constant can cause an accidental identifier match to outrank a more relevant product, while a small value may not change the result at all.

Add business signals without hiding relevance

WooCommerce agencies commonly need to promote available, popular, or commercially preferred products. These signals should be added after establishing a reliable lexical baseline. For example, a rank profile can combine BM25 with normalized sales or margin signals.

rank-profile bm25_with_business_signals {<br>    first-phase {<br>        expression: 0.80 * bm25(name)<br>                  + 0.25 * bm25(brand)<br>                  + 0.15 * bm25(description)<br>                  + 0.10 * attribute(popularity_score)<br>                  + 0.05 * attribute(margin_score)<br>    }<br>}

The numbers above are starting points, not universal recommendations. BM25 and business features may have different numerical ranges, so inspect score distributions before assigning weights. Normalized attributes, bounded functions, or a second-phase expression can make the combination more predictable.

Do not use popularity to compensate for missing textual relevance. A highly popular product that does not match the query should not routinely outrank a product that clearly matches the customer’s terms. If a promotion must override relevance, make that behavior explicit with a campaign rule, a query-time boost, or a curated result.

Handle filters and facets separately

Color, size, price range, brand, category, and stock status are usually structured constraints rather than free text. Store them as attributes or structured fields so the WooCommerce storefront can filter and aggregate them efficiently. A customer query such as running shoes can then be ranked with BM25 while filters such as size 10 and color black restrict the candidate set.

Keep the displayed facet values synchronized with the indexed product state. If a variation is out of stock but the parent product remains searchable, define whether the facet represents the parent catalog item or currently purchasable variations. This decision affects both the query filter and the data shown in the storefront.

Evaluate BM25 with real WooCommerce queries

Before tuning weights, create a test set from site search logs and support tickets. Include product names, generic category queries, brand queries, SKU searches, misspellings, attribute combinations, and queries with no exact match. For each query, record the products that should appear in the first page and the products that must not appear.

  1. Run the same test set with a simple BM25 profile.
  2. Check whether product names, brands, and descriptions contribute in the expected order.
  3. Measure top-result accuracy and inspect failures manually.
  4. Add exact identifier handling and business signals only where the baseline needs improvement.
  5. Compare each change against the saved test set before deploying it.

Use Vespa query tracing and ranking explanations when a result is surprising. The goal is to determine whether the problem comes from tokenization, missing indexed content, an incorrect filter, a field weight, or a business signal. Changing a coefficient without identifying the underlying cause often produces fragile search behavior.

Practical deployment guidance

Keep rank profiles versioned with the schema and deploy changes through the same process used for application and indexing updates. When changing field definitions or enabling BM25 on new fields, reindex the catalog as required by the schema change. During rollout, compare the existing WooCommerce search results with the Vespa results for the same query and filters.

For agencies managing several stores, make field weights and business-signal coefficients configurable per tenant rather than copying unrelated rank profiles. The same BM25 structure can support multiple catalogs, while each store retains control over brand emphasis, promotional signals, and inventory rules.

Trending posts
You might also like