WPSolr logo

Table of contents :

Importing a WooCommerce Catalog Into Weaviate

100-days-of-weaviate-012

Table of contents :

A WooCommerce catalog is a practical first dataset for a product search, recommendation, or shopping assistant built with Weaviate. The import needs to preserve structured commerce fields such as SKU, price, stock status, categories, attributes, and product URLs while also creating searchable text from the product name and description.

For an agency, the importer should be repeatable. It should handle pagination, avoid duplicate objects, support incremental updates, and keep Weaviate in sync when products are edited or deleted in WooCommerce.

Plan the product representation

Before importing products, decide which WooCommerce fields should be stored as properties and which should contribute to the vectorized text. A useful product object commonly includes:

  • Identity: WooCommerce product ID, SKU, slug, and product URL.
  • Commerce data: regular price, sale price, currency, stock status, and purchasability.
  • Catalog structure: categories, tags, brands, attributes, and variation information.
  • Content: name, short description, full description, and a normalized search text field.
  • Operational data: source site, last modified timestamp, and import version.

Keep values that must be filtered or displayed as typed properties. For example, store prices as numbers, stock status as text, and product IDs as integers or strings. Do not put the entire WooCommerce API response into one unstructured text property and expect filtering to work reliably.

Create a Weaviate collection

The following example uses the Weaviate Python client version 4 API. The collection is configured with a text vectorizer, while identifiers and commerce fields remain available for filtering and retrieval.

import weaviate
from weaviate.classes.config import Configure, DataType, Property

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=WEAVIATE_URL,
    auth_credentials=weaviate.auth.AuthApiKey(WEAVIATE_API_KEY),
)

products = client.collections.create(
    name="WooProduct",
    vector_config=Configure.Vectors.text2vec_weaviate(),
    properties=[
        Property(name="wooId", data_type=DataType.INT),
        Property(name="sku", data_type=DataType.TEXT),
        Property(name="name", data_type=DataType.TEXT),
        Property(name="slug", data_type=DataType.TEXT),
        Property(name="description", data_type=DataType.TEXT),
        Property(name="searchText", data_type=DataType.TEXT),
        Property(name="categories", data_type=DataType.TEXT_ARRAY),
        Property(name="attributes", data_type=DataType.TEXT_ARRAY),
        Property(name="price", data_type=DataType.NUMBER),
        Property(name="stockStatus", data_type=DataType.TEXT),
        Property(name="permalink", data_type=DataType.TEXT),
        Property(name="modifiedGmt", data_type=DataType.DATE),
    ],
)

Choose the vectorizer according to the deployment requirements of the project. If the catalog uses an external embedding model, configure that model in Weaviate and import the same properties. The important design principle is to make searchText readable and useful rather than relying only on an HTML description.

Fetch the WooCommerce catalog page by page

The WooCommerce REST API returns products in pages. The per_page value is limited by the API, and the response headers expose the total number of pages. Use HTTPS, a dedicated API key, and least-privilege credentials for the store.

import requests


def fetch_products(store_url, consumer_key, consumer_secret):
    page = 1

    while True:
        response = requests.get(
            f"{store_url}/wp-json/wc/v3/products",
            auth=(consumer_key, consumer_secret),
            params={
                "page": page,
                "per_page": 100,
                "status": "publish",
                "orderby": "id",
                "order": "asc",
            },
            timeout=30,
        )
        response.raise_for_status()

        products = response.json()
        if not products:
            break

        yield from products

        total_pages = int(response.headers.get("X-WP-TotalPages", page))
        if page >= total_pages:
            break
        page += 1

For large catalogs, add retry handling for transient HTTP failures and rate limiting. Logging the page number, product count, and last successfully imported product makes a scheduled job much easier to operate.

Normalize WooCommerce products

WooCommerce descriptions often contain HTML, and category or attribute values are nested objects. Normalize them before sending the data to Weaviate. Preserve the original permalink so search results can link back to the store.

import re
from html import unescape


def strip_html(value):
    text = re.sub(r"<[^>]+>", " ", value or "")
    return " ".join(unescape(text).split())


def normalize_product(product):
    name = product.get("name", "").strip()
    short_description = strip_html(product.get("short_description"))
    description = strip_html(product.get("description"))

    categories = [item["name"] for item in product.get("categories", [])]
    attributes = []
    for attribute in product.get("attributes", []):
        attribute_name = attribute.get("name", "")
        options = ", ".join(attribute.get("options", []))
        if attribute_name and options:
            attributes.append(f"{attribute_name}: {options}")

    search_text = " | ".join(
        part for part in [
            name,
            short_description,
            description,
            "Categories: " + ", ".join(categories),
            "Attributes: " + "; ".join(attributes),
        ]
        if part
    )

    price_text = product.get("sale_price") or product.get("regular_price") or "0"

    return {
        "wooId": int(product["id"]),
        "sku": product.get("sku") or "",
        "name": name,
        "slug": product.get("slug") or "",
        "description": description,
        "searchText": search_text,
        "categories": categories,
        "attributes": attributes,
        "price": float(price_text or 0),
        "stockStatus": product.get("stock_status") or "unknown",
        "permalink": product.get("permalink") or "",
        "modifiedGmt": product.get("date_modified_gmt"),
    }

In production, decide how to represent variable products. One option is to index the parent product and include variation attributes in the text. Another is to index each variation as a separate object with a parent product ID. The second approach supports precise size, color, or material filtering but requires a clear result-display strategy.

Use deterministic UUIDs for repeatable imports

A rerun should update an existing Weaviate object instead of creating a second copy. Generate a UUID from a stable namespace and the WooCommerce product ID. Including the store identifier is important when one Weaviate instance contains catalogs from multiple sites.

import uuid

SOURCE_NAMESPACE = uuid.UUID("6f0c3d8e-0d5f-4f7c-8c0c-0d6e0f8f4e22")


def product_uuid(site_key, woo_id):
    return str(uuid.uuid5(SOURCE_NAMESPACE, f"{site_key}:product:{woo_id}"))

Do not derive the UUID from a product name or permalink if those values can change. A product ID is stable within a WooCommerce site and is a better synchronization key.

Batch import the catalog

Batching reduces network overhead and lets Weaviate process multiple objects efficiently. The exact batch helper can vary with the Python client version, so keep the import logic isolated behind a small function that can be tested independently.

from weaviate.util import generate_uuid5


def import_catalog(client, product_records, site_key):
    collection = client.collections.use("WooProduct")

    with collection.batch.fixed_size(batch_size=100) as batch:
        for product in product_records:
            properties = normalize_product(product)
            batch.add_object(
                properties=properties,
                uuid=product_uuid(site_key, properties["wooId"]),
            )

            if batch.number_errors > 0:
                print("Batch errors:", batch.number_errors)

After the batch closes, inspect failed objects rather than treating the job as successful solely because the HTTP request completed. Record the WooCommerce ID, error message, and import timestamp for each failure. A product with malformed data should not prevent the rest of the catalog from loading.

Choose an update strategy

For an initial import, a full catalog load is straightforward. For ongoing synchronization, combine one or more of these approaches:

  • Scheduled delta imports: request products modified after the last successful timestamp and upsert them using deterministic UUIDs.
  • WooCommerce webhooks: subscribe to product created, updated, and deleted events, then enqueue the corresponding synchronization task.
  • Periodic reconciliation: run a complete ID comparison daily or weekly to detect missed webhooks and deleted products.

WooCommerce webhook payloads should normally trigger a job rather than performing a full Weaviate write inside the webhook request. Queue processing gives the worker time to fetch the current product, handle retries, and avoid duplicate work.

For deletions, maintain the same deterministic UUID and call the Weaviate delete operation when WooCommerce reports that a product no longer exists or has been permanently removed. If products are merely unpublished, keeping them in Weaviate with a published property can be useful for auditability, but every search query must then filter out unpublished records.

Query with semantic search and filters

Catalog search usually needs both semantic relevance and structured constraints. A customer may search for “lightweight waterproof hiking jacket” and also require a particular price range or stock status.

results = products.query.hybrid(
    query="lightweight waterproof hiking jacket",
    filters=(
        products.filter.by_property("stockStatus").equal("instock")
        & products.filter.by_property("price").less_or_equal(200)
    ),
    limit=10,
    return_properties=[
        "name",
        "price",
        "categories",
        "permalink",
        "stockStatus",
    ],
)

for item in results.objects:
    print(item.properties["name"], item.properties["permalink"])

Hybrid search is often a good default for WooCommerce because product names, SKUs, brand terms, and exact model numbers benefit from keyword matching, while descriptive queries benefit from vectors. Use filters for business rules such as availability, catalog visibility, price, brand, and site ownership.

Operational checks for agency deployments

  • Store the WooCommerce site key and source product ID on every object.
  • Track the last successful page, modification timestamp, and batch failure count.
  • Use deterministic UUIDs so reruns are safe and idempotent.
  • Keep secrets in environment variables or a managed secrets service, never in product properties or source control.
  • Test descriptions containing HTML, empty prices, missing SKUs, Unicode characters, and products with many variations.
  • Compare the WooCommerce published product count with the searchable Weaviate count after a full reconciliation.
  • Close the Weaviate client cleanly in scripts and workers after the import completes.

A reliable catalog pipeline separates extraction, normalization, loading, and synchronization. That separation allows an agency to reuse the importer across stores while changing only the WooCommerce credentials, field mappings, collection configuration, and search filters for each client.

Trending posts
Recommendations powered by WPSOLR and  Recombee
You might also like
Recommendations powered by WPSOLR and  Recombee