Large WooCommerce catalogs should not be loaded into Vespa with one request per product and no coordination. That approach is slow, difficult to resume, and likely to trigger rate limits on the WooCommerce side. A reliable import treats catalog loading as a controlled data pipeline with checkpoints, bounded concurrency, idempotent document IDs, and a clear recovery strategy.
Define the document model before importing
Start by deciding which WooCommerce fields belong in the Vespa document. Search usually needs the product name, SKU, descriptions, categories, attributes, stock status, price, visibility, images, and variation data. Do not copy the entire WooCommerce response into every searchable field without a plan. Keep source data that may be useful for debugging or reprocessing, but map it deliberately to the Vespa schema.
A product document might contain fields such as:
id: the stable WooCommerce product IDsku: the merchant-facing stock-keeping unittitle: the searchable product namedescription: normalized text extracted from the product descriptioncategories: category names or category IDsattributes: flattened searchable attributesprice: a numeric value for sorting and filteringstock_status: a categorical field for availability filtersimage_url: the preferred product imageupdated_at: the source modification timestamp
Use a deterministic document ID, for example product::<woocommerce-product-id>. The same source product must always address the same Vespa document. This makes retries safe and prevents duplicate documents when an import is interrupted.
Read WooCommerce in pages
For an initial import, use the WooCommerce REST API with a fixed page size that the store can handle. Request only the fields and embedded resources required by the transformation step where possible. Capture pagination metadata, including the total number of pages when it is available, and persist the last successfully processed page or product ID.
For very large stores, avoid depending on a single long-running request sequence. A coordinator can divide the catalog into ranges or page assignments and place those assignments on a queue. Workers then fetch and transform their assigned records. The coordinator should record the source query, page range, start time, completion time, and number of products processed.
Be careful when the catalog changes during the initial load. Offset-based pagination can shift when products are inserted or deleted. Prefer a stable export or a query ordered by a durable field such as product ID or modification time. If the source API does not provide a safe cursor, run a change reconciliation after the bulk load using WooCommerce webhooks, modified timestamps, or a second scan.
Transform source records into feed operations
Keep WooCommerce-specific transformation logic outside the Vespa feed client. A transformation function should normalize HTML, remove fields that are not useful for search, convert prices to a consistent numeric representation, and produce the exact document fields expected by the Vespa schema.
For example, a normalized operation can be represented internally as:
{
"put": "id:catalog:product::8421",
"fields": {
"title": "Merino Wool Jacket",
"description": "Lightweight jacket for outdoor use",
"sku": "MWJ-8421",
"categories": ["Jackets", "Outdoor"],
"price": 129.95,
"stock_status": "instock",
"source_updated_at": "2025-02-14T10:32:00Z"
}
}The exact feed representation depends on the Vespa client and document API being used. The important properties are stable IDs, schema-compatible values, and a transformation that can be run repeatedly with the same result.
Feed in bounded batches
Do not build one enormous in-memory array. Fetch a page, transform its records, feed a bounded batch, wait for the result, and then release the memory. The correct batch size depends on document size, network latency, container resources, and indexing load. Begin conservatively, measure throughput and latency, and increase the batch size only when the system remains healthy.
Use controlled concurrency rather than creating an unbounded task for every product. A typical importer has a producer that reads WooCommerce pages and a fixed number of feed workers. A bounded queue between them provides backpressure: when Vespa slows down, the importer stops fetching more source data instead of exhausting memory or overwhelming the store.
Conceptually, the pipeline looks like this:
- Read a page or stable range from WooCommerce.
- Validate and normalize each product.
- Place feed operations on a bounded queue.
- Send operations with a limited number of workers.
- Record successful and failed operations.
- Advance the checkpoint only after the relevant batch has been acknowledged.
Handle failures without corrupting progress
A bulk importer needs to distinguish transient failures from permanent data errors. Connection resets, temporary service unavailability, and overloaded feed endpoints can usually be retried with exponential backoff and jitter. Invalid field types, missing required fields, and schema mismatches need to be recorded for correction rather than retried forever.
Every batch should produce an outcome containing at least the source identifier, document ID, operation type, response status, error message, and retry count. Store failed operations in a dead-letter location such as object storage or a durable queue. This lets the main import finish while preserving enough information to reprocess individual products later.
Retries must be idempotent. Repeating a put operation for the same document ID should update that document rather than create another copy. If the importer supports deletes, use the same deterministic ID and record deletion operations separately so that a failed delete cannot be mistaken for a successful update.
Use full replacements and partial updates intentionally
During a first load, a full document replacement is often easier to reason about because the document contains a complete representation of the WooCommerce product. For later synchronization, partial updates can be useful when only inventory or price changes frequently. Make sure that a partial update cannot leave stale fields behind when a product loses a category, attribute, image, or variation.
For example, a stock-only update should change the availability fields without accidentally replacing the title or description with an incomplete source record. Conversely, when a product is deleted or made permanently unavailable, define whether it should be removed from search, marked as hidden, or retained for historical reporting.
Plan for variations and visibility rules
WooCommerce variable products require an explicit indexing decision. You can index only the parent product, index each variation as a separate searchable document, or index the parent with denormalized variation data. Index variations separately when customers search or filter by variation-specific SKU, price, size, color, or stock status. Use a parent reference so the application can group results and display the correct product page.
Apply WooCommerce visibility rules during transformation. Products excluded from catalog or search visibility should not become searchable merely because they were returned by an administrative API request. Likewise, account-specific prices, restricted products, and region-specific availability may require separate document fields or query-time filtering rather than a single global product representation.
Reconcile after the initial feed
A successful feed response does not prove that the catalog is complete. After the bulk operation, compare the set of source product IDs with the set of indexed IDs, or run a source query for products changed during the import window. This catches products added, deleted, or modified while the initial load was running.
Useful validation checks include:
- the number of WooCommerce products read versus the number of feed operations acknowledged
- the number of permanent failures and their error categories
- the number of indexed documents by product type and visibility state
- the percentage of documents with missing required fields
- sample searches for popular products, SKUs, categories, and out-of-stock items
- comparison of price and stock values for a random sample of products
Do not declare the import complete until failed operations have been retried or explicitly excluded and the reconciliation checks have passed.
Make subsequent synchronization cheaper
The initial bulk load and ongoing synchronization should be separate workflows. After the catalog is loaded, consume WooCommerce webhooks for product creation, updates, deletions, and stock changes where they provide sufficient coverage. Treat webhook delivery as a notification, not as the complete source record: fetch the current product by ID before indexing it, and make the handler idempotent.
Webhooks can be delayed, duplicated, or missed. Run a periodic reconciliation job using modification timestamps or an incremental export. This job repairs drift without repeatedly reloading the entire catalog.
Operational controls for agencies
Expose import settings rather than hard-coding them into a project-specific script. Useful settings include page size, feed batch size, worker count, request timeout, retry limit, backoff range, checkpoint location, and whether the run performs puts, deletes, or both.
Monitor both sides of the pipeline. On WooCommerce, watch API response times, error rates, authentication failures, and server load. On Vespa, watch feed latency, rejected operations, container resource usage, document counts, and query performance. Add a dry-run mode that fetches and validates records without sending them, and a sample mode that processes a limited number of pages before a production import.
A repeatable importer should be safe to stop and restart. With stable document IDs, durable checkpoints, bounded concurrency, detailed error records, and a reconciliation pass, agencies can load large WooCommerce catalogs predictably without turning the store or the search cluster into a bottleneck.