In an Unstructured-based content pipeline, partitioning and chunking solve different problems. Partitioning converts a source document into meaningful elements such as titles, paragraphs, lists, tables, images, and page breaks. Chunking then combines those elements into retrieval-sized text units for embeddings, vector search, or large-language-model prompts.
Keeping these stages separate is especially important for WooCommerce agencies. Product catalogs, policy pages, order guides, technical documentation, and support exports have different structures. If a pipeline treats every source as a plain string, it can easily split a shipping table from its heading, mix unrelated product sections, or produce chunks that are too large for reliable retrieval.
What partitioning does in Unstructured
Partitioning is the document-understanding stage. Functions such as partition_pdf, partition_html, and partition_docx inspect a file or document and return a sequence of typed elements. Depending on the input and available dependencies, the result may include Title, NarrativeText, ListItem, Table, Image, and other element types.
Each element can carry metadata, including the source filename, page number, coordinates, category, and parent or section information. That metadata is valuable when a WooCommerce support assistant needs to cite the page containing a returns rule or when an agency needs to reprocess only the pages changed in a client manual.
from unstructured.partition.html import partition_html
elements = partition_html(filename="shipping-policy.html")
for element in elements:
print(element.category, element.text[:80], element.metadata.page_number)The output from partitioning is not necessarily ready for vector search. A single product guide may produce many small elements, including a heading followed by several paragraphs. Embedding each element independently can lose context, while sending the entire document to a model can exceed token limits.
What chunking does in Unstructured
Chunking groups partitioned elements into larger units while attempting to preserve document structure. A chunk usually contains the text that will be embedded or supplied as context to a downstream model. Chunk metadata can retain useful source information and can identify the original elements that contributed to the chunk.
Unstructured provides chunking utilities such as chunk_elements and chunk_by_title. The basic approach focuses on size constraints. The title-based approach uses title elements as section boundaries, which is often a better fit for policy documents, product documentation, and knowledge bases organized with headings.
from unstructured.chunking.title import chunk_by_title
from unstructured.partition.pdf import partition_pdf
elements = partition_pdf(filename="store-handbook.pdf")
chunks = chunk_by_title(
elements,
max_characters=3000,
new_after_n_chars=2400,
combine_text_under_n_chars=500,
)
for chunk in chunks:
print(chunk.text)The exact settings should be tested against the documents being indexed. max_characters places an upper bound on chunk size, while new_after_n_chars encourages a new chunk once the preferred size is reached. A smaller value can improve retrieval precision, but it may also separate closely related instructions.
Partitioning is semantic; chunking is retrieval-oriented
A useful mental model is that partitioning answers, “What are the meaningful parts of this document?” Chunking answers, “Which parts should be retrieved and presented together?”
- Partitioning: identifies structure and element types.
- Chunking: groups elements within practical size limits.
- Embedding: converts each final chunk into a vector.
- Retrieval: selects relevant chunks for a search query or assistant response.
For example, a WooCommerce returns policy might be partitioned into a title, several paragraphs, a numbered list, and a table of product categories. Chunking can keep the title with the following policy text and, where possible, retain the list or table as a coherent unit. The embedding should represent a complete rule rather than an isolated sentence such as “within 30 days.”
Choosing a strategy for WooCommerce content
Use title-based chunking for documentation and policies
chunk_by_title is a strong default for store handbooks, shipping policies, payment documentation, and implementation guides. Headings commonly represent stable topics such as “International Shipping,” “Refund Eligibility,” or “Subscription Renewals.” Keeping content under those headings together improves both search relevance and answer traceability.
Use basic chunking for irregular or weakly structured sources
Some exports have few headings or contain long narrative elements. In those cases, basic element chunking can provide predictable size control. It is also useful as a fallback for content assembled from email exports, plain text files, or OCR output where heading detection is unreliable.
Handle tables deliberately
Tables need special attention in commerce workflows. A table containing shipping zones, tax rules, or product compatibility may be highly relevant to a query, but flattening it without headers can make the values ambiguous. Preserve table elements when possible, include the surrounding heading, and validate how the partitioner represents rows and columns before indexing.
Chunk size, overlap, and metadata
There is no universal chunk size. A practical starting point for a WooCommerce knowledge base is to test chunks between roughly 500 and 1,500 words, or the equivalent token range required by the embedding model. Short FAQ answers may work best as individual chunks, while installation guides may need larger section-based chunks.
Overlap can help when a sentence or instruction crosses a boundary, but excessive overlap increases indexing cost and can cause duplicate search results. Prefer structure-aware boundaries first. Add overlap only when evaluation shows that relevant context is regularly being split.
Keep metadata such as the source URL, document ID, page number, section title, language, store identifier, and content version. For a multi-client agency platform, a store_id or tenant identifier is essential so that retrieval filters cannot return one merchant’s policies to another merchant.
record = {
"text": chunk.text,
"metadata": {
"store_id": "client-247",
"source": "shipping-policy.pdf",
"section": chunk.metadata.section,
"page_number": chunk.metadata.page_number,
"content_version": "2025-02-14"
}
}Common implementation mistakes
- Embedding partitioned elements without testing context: headings and short list items may be too small to retrieve effectively.
- Chunking raw text before partitioning: character-based splitting can cut through tables, headings, and numbered instructions.
- Assuming every format produces the same elements: PDF, HTML, DOCX, and OCR inputs can have different detection quality and metadata.
- Ignoring updates: unchanged chunks should remain stable where possible, while modified documents should be re-partitioned and re-indexed with a new version.
- Failing to inspect output: save representative partitioned elements and final chunks during development so that parsing errors are visible before customer-facing deployment.
A practical evaluation workflow
For each important WooCommerce content type, create a small test set of real queries. Include questions about shipping destinations, refund windows, product specifications, subscription cancellation, and operational procedures. Compare retrieval results using raw partitioned elements, basic chunks, and title-based chunks.
Measure whether the correct chunk appears in the top results, whether the chunk contains enough surrounding context to answer accurately, and whether citations point to a useful source location. Also inspect cost, indexing time, duplicate results, and the number of chunks generated per document.
Partitioning should be treated as the source of structured truth, while chunking should be treated as a configurable representation for a particular retrieval system. Retaining the original elements allows an agency to change chunk sizes, embedding models, or retrieval strategies without repeatedly losing the structure of the source document.