Basic Chunking with Unstructured

WooCommerce search and retrieval systems usually receive content in uneven shapes: product descriptions, attribute lists, shipping policies, order documentation, support pages, and exported catalog files. Before embedding that content, an agency needs a predictable way to divide it into retrieval-sized pieces. Unstructured provides a straightforward pipeline for doing this: partition a document into typed elements, then combine those elements into chunks with controlled size and overlap.

Partitioning and chunking are different operations

Unstructured first parses a source document into elements such as Title, NarrativeText, ListItem, Table, and Image. Partitioning is responsible for identifying those elements. Chunking is the later step that combines or splits element text so it fits the limits of an embedding model or retrieval system.

This distinction matters for WooCommerce projects. A product description should not be treated as an undifferentiated block if the parser can preserve headings, lists, and tables. At the same time, sending every short element as an individual vector can create many tiny, low-context records. Basic chunking provides a practical middle ground.

Installing Unstructured

For common HTML, text, and document workflows, install the core package and the partitioning dependencies required by the file types in your pipeline:

pip install "unstructured[all-docs]"

For a production agency project, it is usually better to choose only the extras required by the supported source formats. A site that imports HTML product content does not necessarily need every document-processing dependency.

Basic chunking with Python

The following example partitions a WooCommerce product guide and then groups the resulting elements into chunks. The values are examples rather than universal defaults.

from unstructured.partition.auto import partition
from unstructured.chunking.basic import chunk_elements

# The input could be an HTML, PDF, DOCX, or text file.
elements = partition(filename="woocommerce-product-guide.html")

chunks = chunk_elements(
    elements,
    max_characters=1200,
    new_after_n_chars=1000,
    overlap=150,
)

for index, chunk in enumerate(chunks):
    print({
        "chunk_index": index,
        "text": chunk.text,
        "metadata": chunk.metadata.to_dict(),
    })

max_characters is the hard upper limit used by the basic chunker. new_after_n_chars is a soft target that encourages a new chunk before the hard limit is reached. overlap repeats a limited amount of text between adjacent chunks, which can help when an answer depends on context near a chunk boundary.

The exact useful values depend on the embedding model, the language of the catalog, and the expected answer size. Character counts are not token counts, so validate the resulting chunks against the tokenizer used by the selected embedding and language models.

What basic chunking does well

It is simple to add to an existing Unstructured partitioning pipeline.
It works across content that has already been converted into Unstructured elements.
It allows an agency to set a predictable maximum size for vector records.
It can preserve useful source metadata attached during partitioning.
It is suitable for initial indexing of mixed operational content such as policies, guides, and support documentation.

Basic chunking processes elements in sequence. It is not a WooCommerce-aware splitter: it does not know that a product SKU, price, variation, or shipping restriction may need to remain together. Domain metadata and content preparation still belong in the ingestion pipeline.

Preparing WooCommerce content before partitioning

When indexing product or store content, create a clean source representation before handing it to Unstructured. For example, an HTML product document might include the product name, SKU, short description, full description, attributes, stock status, and shipping notes in a consistent order.

<h1>Trail Running Shoes</h1>
<p>SKU: TR-001</p>
<h2>Description</h2>
<p>Lightweight shoes for mixed-terrain running.</p>
<h2>Attributes</h2>
<ul>
  <li>Material: Mesh and rubber</li>
  <li>Sizes: 7–13</li>
</ul>
<h2>Shipping</h2>
<p>Ships within two business days.</p>

Keeping the product identifier in the source text can make retrieved results easier for a language model to interpret. It should also be stored as structured metadata, such as product_id, sku, site_id, and language, so the application can filter or display results reliably.

Attaching application metadata

Metadata produced by a parser describes the source element, but a WooCommerce integration normally needs additional application-level fields. Add those fields when converting chunks to records for the vector database.

records = []

for index, chunk in enumerate(chunks):
    parser_metadata = chunk.metadata.to_dict()

    records.append({
        "id": f"product-guide:{index}",
        "text": chunk.text,
        "metadata": {
            **parser_metadata,
            "site_id": "store-001",
            "content_type": "product_guide",
            "source_url": "https://example.com/product-guide/",
            "product_id": "TR-001",
            "sku": "TR-001",
            "chunk_index": index,
        },
    })

Use a deterministic identifier when possible. A common pattern is a stable source identifier plus the chunk index, or a hash of the canonical source and chunk text. Stable IDs make it possible to delete and re-index a product without leaving stale vectors behind.

Handling tables and product attributes

Product specifications are often represented as tables. Basic chunking can include table elements in the sequence, but the downstream embedding quality depends on how the table was partitioned and serialized. Test whether the resulting text preserves the relationship between field names and values.

For retrieval, a normalized representation can be clearer than raw table markup:

Product: Trail Running Shoes
SKU: TR-001
Material: Mesh and rubber
Water resistance: Water-repellent
Available sizes: 7, 8, 9, 10, 11, 12, 13

For highly structured catalogs, store important fields in database columns or vector metadata as well as in the embedded text. A vector search should not be the only mechanism used to enforce exact SKU, stock, price, or availability filters.

Choosing overlap carefully

Overlap can reduce the chance that a sentence split across two chunks loses its context. However, large overlap increases the number of repeated tokens, storage requirements, indexing time, and the probability of duplicate search results.

For product documentation, a small overlap can be useful around explanatory paragraphs. For short product records that already contain their SKU and product name, overlap may provide little benefit. Measure retrieval quality with real agency queries such as “Which size is available for SKU TR-001?” and “What is the delivery time for water-repellent shoes?” rather than selecting a value solely from a rule of thumb.

Basic chunking versus title-aware chunking

Basic chunking is useful when a simple size-based strategy is acceptable. It does not make section headings the primary boundary. If a policy, manual, or catalog guide has meaningful headings and each section should remain together where possible, compare it with Unstructured’s title-aware chunking options, such as chunk_by_title.

Title-aware strategies are not automatically better for every WooCommerce source. Product feeds may have repeated headings, incomplete markup, or one record per file. Evaluate the actual parsed elements and inspect representative chunks before selecting a strategy for all stores in a multi-client platform.

Testing the output before indexing

Inspect chunks as data, not just as a successful function result. A useful validation report should include the number of elements, number of chunks, character-length distribution, empty chunks, unusually large chunks, and representative samples from each source type.

lengths = [len(chunk.text) for chunk in chunks if chunk.text.strip()]

if lengths:
    print("chunks:", len(lengths))
    print("minimum:", min(lengths))
    print("maximum:", max(lengths))
    print("average:", sum(lengths) / len(lengths))

for index, chunk in enumerate(chunks[:3]):
    print(f"--- chunk {index} ---")
    print(chunk.text[:500])

Also test re-indexing behavior, HTML entities, multilingual content, variation data, and documents containing tables or page breaks. A chunk that looks acceptable in a guide may be unsuitable for a product export with thousands of repeated attributes.

Operational considerations for agencies

Pin compatible Unstructured and dependency versions in the ingestion environment.
Record the parser and chunking configuration with each indexing job.
Keep source identifiers and URLs so retrieved content can be traced to the store.
Re-index changed products using stable IDs and remove records for deleted products.
Apply access controls before retrieval when multiple client stores share infrastructure.
Do not treat chunk text as authoritative for live price, inventory, tax, or order data.

Basic chunking with Unstructured is a useful baseline for WooCommerce knowledge retrieval: parse documents into meaningful elements, combine them within explicit size limits, preserve application metadata, and validate the output against real store queries before embedding the records.

Table of contents :