Why tokenization can quietly damage WooCommerce search
When a shopper searches for a product, the visible query is only the starting point. Before Weaviate can compare the query with indexed content, it breaks text into tokens. Those tokens determine which terms are available to lexical search, including the BM25 component of a hybrid query.
This makes tokenization a search-quality issue that is easy to miss. A product title can look highly relevant to a merchandiser while producing weak lexical matches because the query and the indexed value were split differently.
For WooCommerce agencies, this is especially important for product names, variation labels, brand names, model numbers, dimensions, and technical specifications. A small punctuation or casing difference can change the search behavior even when the two strings appear equivalent to a customer.
What tokenization controls in Weaviate
Tokenization is configured per text property. Common options include:
- word: Splits text into words and normalizes case. This is generally suitable for product titles, descriptions, and searchable attributes.
- whitespace: Splits values at whitespace boundaries. Punctuation remains more significant, which can be useful for some identifiers but can also create unexpected matches.
- lowercase: Treats the lowercased value as one token rather than splitting it into separate words.
- field: Treats the complete value as one token, preserving the value as a single searchable unit.
- trigram: Indexes character trigrams, which can help with partial and typo-tolerant matching at the cost of a larger index and different scoring behavior.
The exact choice affects lexical matching and filters that depend on the indexed representation. It does not automatically change the vector generated by a vectorizer. Hybrid search can therefore expose an imbalance: the vector component may recognize that two product names are related while the BM25 component fails to match important terms.
Common WooCommerce failure patterns
Hyphens and compound product names
Consider a catalog containing these values:
USB-C ChargerUSB C ChargerUSBC Charger
A shopper may treat these as the same product family. Depending on the tokenization used and the exact query, the lexical component may produce different tokens for each form. A vector search may still retrieve related products, but the ranking can change substantially when the hybrid balance gives BM25 meaningful weight.
SKUs and model numbers
Identifiers such as ABC-123-XL, ABC123XL, and ABC 123 XL are not ordinary prose. Splitting them into words may improve some partial searches and damage exact lookup behavior for others.
Do not force one property to serve both product discovery and exact identifier lookup. Store the SKU as a dedicated structured property, and expose a separate normalized search field when customers commonly enter alternate forms.
Dimensions, sizes, and technical specifications
Values such as 27-inch 4K, 27" 4K, and 4K 27 inch can be semantically equivalent but lexically different. Unit symbols, punctuation, and number-letter combinations deserve explicit normalization before indexing.
Brand and casing differences
A catalog may contain iPhone, IPHONE, and iphone. A case-insensitive word strategy is usually appropriate for shopper-facing product text, but an agency should not assume that every tokenization mode normalizes case in the same way.
Choosing tokenization by property
Use different properties for different search jobs instead of applying one global rule to every WooCommerce field.
- Product title: Usually start with
word. It supports ordinary phrase-like queries and gives BM25 useful individual terms. - Long description: Usually use
word, while removing boilerplate and repeated specification blocks from the indexed content. - SKU: Keep a dedicated identifier property. Add a normalized alternate value if searches commonly omit hyphens or spaces.
- Brand: Use a consistent canonical value and consider a separate exact or filterable property for brand facets.
- Short technical codes: Test
field,lowercase, or a normalized companion field when the complete value should behave as one unit. - Autocomplete or partial matching: Evaluate
trigramonly after measuring index size, latency, and ranking quality against the catalog.
Example collection configuration
With the Weaviate Python client, tokenization is selected when the collection and property are created. The following example uses ordinary word tokenization for the title and a separate field for a normalized SKU.
from weaviate.classes.config import Configure, Property, DataType, Tokenization
products = client.collections.create(
name="Product",
vector_config=Configure.Vectors.text2vec_openai(),
properties=[
Property(
name="title",
data_type=DataType.TEXT,
tokenization=Tokenization.WORD,
index_searchable=True,
),
Property(
name="skuNormalized",
data_type=DataType.TEXT,
tokenization=Tokenization.FIELD,
index_searchable=True,
index_filterable=True,
),
Property(
name="brand",
data_type=DataType.TEXT,
tokenization=Tokenization.WORD,
index_searchable=True,
index_filterable=True,
),
],
)
The precise client syntax can vary by Weaviate client version, so verify the configuration API used by the deployment. The design principle is more important than the example: shopper-facing text and exact identifiers should not be forced into the same representation.
Normalize before indexing
Tokenization is not a substitute for catalog normalization. During the WooCommerce synchronization process, generate canonical fields for values that customers enter inconsistently.
def normalize_sku(value: str) -> str:
return "".join(character for character in value.upper() if character.isalnum())
product = {
"title": "USB-C Charger 65W",
"sku": "CHG-65W-USB-C",
"skuNormalized": normalize_sku("CHG-65W-USB-C"),
}
Keep the original SKU for display and order processing. Use the normalized value for lookup only if that behavior is intentional. The normalization function should be documented and shared by both the indexing pipeline and the query preprocessing code; otherwise, a normalized index will still fail when the incoming query is left untouched.
How tokenization interacts with hybrid search
A hybrid query combines vector similarity with keyword relevance. If the query is wireless noise cancelling headphones, the vector component may identify semantically related products even when exact terms differ. The BM25 component rewards indexed terms that overlap with the query.
Tokenization changes the BM25 side of that calculation. A poorly chosen configuration can cause:
- Exact product terms to be split into unusable fragments.
- Important words to remain attached to punctuation.
- Short identifiers to receive weak or surprising scores.
- Common words to dominate results because the meaningful product code was not matched.
- Different catalog records to rank inconsistently despite having equivalent normalized values.
When investigating ranking, test the lexical and vector behaviors separately before changing the hybrid balance. Increasing the vector weight may hide a tokenization problem rather than fix it.
A practical diagnostic workflow
- Collect real queries. Use WooCommerce search logs, zero-result queries, click data, and search-to-cart reports. Include SKU searches and queries containing punctuation.
- Compare raw values. For each failed query, inspect the exact product title, SKU, brand, and attribute values sent to Weaviate.
- Test tokenization variants. Create a small representative collection or staging index using
word,whitespace, and a normalized companion property where appropriate. - Run lexical-only tests. Compare BM25 results before evaluating hybrid results. This isolates indexing behavior from embedding behavior.
- Measure business outcomes. Track zero-result rate, first-result relevance, click-through, add-to-cart rate, and exact-SKU success rate.
- Reindex deliberately. Tokenization is part of the collection schema. Changing it generally requires a migration or replacement collection followed by a complete data import.
Schema decisions agencies should document
For every searchable WooCommerce property, document the source field, normalization rules, tokenization mode, whether it is used for BM25, whether it is used for vectorization, and whether it supports filtering. This prevents a later integration change from silently altering search behavior.
Also record which fields are authoritative. For example, a displayed SKU may retain punctuation, while skuNormalized is generated for lookup. A product title may be localized and customer-facing, while a separate taxonomy or attribute field provides stable filtering.
Tokenization is a low-level indexing choice with a high-level merchandising impact. Treat it as part of the search schema, test it with real WooCommerce queries, and give identifiers and natural-language content separate representations.