Characters, Tokens and Embeddings

When building retrieval-augmented systems for WooCommerce stores, three measurements are easy to confuse: characters, tokens, and embeddings. They describe different stages of the content pipeline, and choosing the wrong one can lead to oversized chunks, incomplete product context, higher API costs, or poor search results.

Characters are a simple size estimate

A character is a letter, number, space, punctuation mark, or other symbol in a text string. Character counts are easy to calculate and useful for initial inspection after a document has been parsed.

For example, a product description might contain 4,800 characters, including HTML text, headings, specifications, and line breaks. That number tells you how large the raw string is, but it does not reliably tell you how much context a language model will receive.

const characterCount = text.length;

Character counts vary significantly by language, formatting, and content type. A string containing mostly ordinary English words may use fewer tokens than a string containing dense technical identifiers, SKU values, URLs, or large quantities of punctuation. Product data can also include repeated symbols, measurements, currency values, and markup that change the relationship between characters and tokens.

Tokens are the model’s processing units

A token is a unit created by a tokenizer before text is sent to a language model or embedding model. A token may represent a complete word, part of a word, punctuation, whitespace, or a sequence of common characters. The exact tokenization depends on the model and tokenizer.

For English prose, a rough estimate is often several characters per token, but this is only a planning approximation. It should not be used as a hard limit. A WooCommerce catalog containing product names such as 12V-DC-Adapter, hexadecimal codes, URLs, or multilingual descriptions may produce a very different token count from ordinary editorial copy with the same character length.

Token limits matter when sending chunks to a chat model or an embedding endpoint. A chunk that is acceptable by character count may exceed the model’s context or request limit after tokenization. The safest workflow is to tokenize with the same tokenizer, or a compatible tokenizer, used by the downstream model and reserve space for metadata and any prompt text.

Why token counts matter for product content

• Long variable products may combine shared descriptions with many variations and exceed the intended chunk size.
• Tables of dimensions, ingredients, compatibility data, or shipping rules can contain more tokens than their visual length suggests.
• SKU lists and technical specifications may tokenize inefficiently.
• Embedding costs and throughput are generally influenced by the number of input tokens rather than the number of characters.

A practical chunking process can retain a character limit for fast pre-filtering, then apply a tokenizer-based limit before embedding. Character limits are useful for operational safeguards; token limits are the authoritative constraint for model requests.

Embeddings are vectors, not text units

An embedding is a numerical vector representing the semantic content of a piece of text. An embedding model receives text, processes its tokens, and returns a vector with a fixed number of dimensions. The vector can then be stored in a vector database and compared with vectors generated from search queries.

Embeddings do not replace characters or tokens, and they do not define how a document should be split. Chunk boundaries are established before embedding. The quality of the resulting vector depends partly on whether the chunk contains a coherent unit of meaning.

For example, these two chunks may have similar token counts but very different retrieval value:

• A complete section describing installation requirements for a product.
• The final sentence of an installation section followed by half of a returns policy.

The second chunk may be technically valid input, but it mixes unrelated subjects. A query about installation could retrieve it because of one sentence while omitting the details needed to answer the customer accurately.

How the three concepts fit together

1. Characters: provide a quick measurement of the parsed string and help enforce basic operational limits.
2. Tokens: determine how much text a model can process and provide the most useful measure for request sizing and cost estimation.
3. Embeddings: convert each selected chunk into a vector for semantic retrieval.

A typical pipeline using parsed WooCommerce documentation or catalog content looks like this:

source document → parsed text → structural chunks → token validation → embeddings → vector index

LlamaParse or another document parser may produce clean text from PDFs, scans, manuals, and other source files. The parser’s output should be inspected before chunking because headings, tables, page breaks, and repeated headers can affect both token counts and semantic quality. A downstream component whose model or tokenizer is not explicitly configured should be treated as undefined until its limits and encoding behavior are verified.

A practical chunking example

Suppose an agency is indexing a manufacturer’s PDF manuals alongside WooCommerce product data. A useful chunk might contain a heading, a short explanation, a specification table, and the product identifier. The chunk should be small enough for the embedding model, but large enough to preserve the relationship between the product and its specifications.

<section><br>  <h2>Installation requirements</h2><br>  <p>This controller requires a 24V DC power supply...</p><br>  <ul><br>    <li>Product family: CTRL-24</li><br>    <li>Operating temperature: -10°C to 50°C</li><br>  </ul><br></section>

Before embedding this content, remove irrelevant navigation, repeated page headers, and parser artifacts. Preserve meaningful labels such as product names, SKUs, section headings, and table headers. If a table is split, repeat the table heading or product context in the following chunk so that the extracted row remains understandable when retrieved independently.

Validation checks for an agency pipeline

• Record character counts for debugging and monitoring.
• Calculate token counts with the intended embedding or language-model tokenizer.
• Set a maximum token threshold below the provider’s hard limit to allow for metadata and future prompt changes.
• Keep chunks semantically focused rather than splitting only at a fixed character interval.
• Store source identifiers, product IDs, page numbers, and headings as metadata instead of relying on the embedding alone.
• Test retrieval with real WooCommerce questions, such as compatibility, installation, warranty, returns, and stock-related queries.
• Re-index content when changing embedding models, because vectors produced by different models are generally not interchangeable.

The key operational rule is straightforward: use characters to observe the raw text, tokens to control model input, and embeddings to support semantic search. Treating these as separate concerns makes it easier to tune chunk sizes, estimate costs, diagnose retrieval failures, and maintain reliable product knowledge systems.