Keeping a Vespa Product Index in Sync with WooCommerce

A Vespa product index is only useful when it reflects the current WooCommerce catalog. Price changes, stock updates, product edits, variations, unpublished products, and deletions must all reach Vespa reliably. For an agency managing several stores, synchronization should be treated as an integration with clear ownership, idempotency, retry handling, and reconciliation rather than as a one-time import.

Choose the document model first

The most important design decision is what a Vespa document represents. A common approach is one document per WooCommerce product, with variations stored as nested data or flattened fields. This works well when customers search for products and the parent product is the result.

For stores where customers search by SKU, variation-level price, availability, or attributes, create one Vespa document per variation. Use a deterministic document ID such as store-42-product-184-variation-207. The ID should not depend on a product name or slug because those values can change.

Whichever model you choose, keep the WooCommerce identifiers in the document:

• store_id: identifies the WooCommerce site in a multi-store deployment.
• product_id: stores the parent product ID.
• variation_id: stores the variation ID when applicable.
• sku: stores the merchant-facing SKU.
• source_modified: records the latest WooCommerce modification timestamp known to the integration.
• visibility and status: support filtering out drafts, private products, and catalog-hidden items.

Use WooCommerce as the source of truth

WooCommerce should remain authoritative for catalog, price, stock, tax, and publication state. Vespa should contain a search-optimized projection of that data. Avoid allowing a search result update to become the only copy of a business-critical value.

A product projection might contain fields such as:

{
  "store_id": "store-42",
  "product_id": 184,
  "variation_id": 207,
  "sku": "BLUE-M",
  "title": "Blue Cotton Shirt",
  "description": "Lightweight cotton shirt",
  "categories": ["Clothing", "Shirts"],
  "attributes": ["color:blue", "size:M"],
  "price": 29.95,
  "regular_price": 34.95,
  "stock_status": "instock",
  "stock_quantity": 12,
  "status": "publish",
  "catalog_visibility": "visible",
  "source_modified": "2025-02-18T10:30:00Z"
}

Do not index formatted HTML from the product description as the only searchable field. Strip presentation markup, preserve useful text, and consider separate fields for title, categories, attributes, and description so Vespa can rank them differently.

Build the initial import as a resumable job

The first synchronization should not depend on live webhooks. Export the catalog through the WooCommerce REST API, process pages in batches, transform each product into the selected Vespa schema, and feed the documents in parallel within controlled limits.

Record progress outside the worker. A useful import record includes the store, endpoint or page cursor, last successful batch, number of documents processed, number of failures, and the import version. If a worker stops after page 37, it should resume from a known checkpoint instead of starting over or silently skipping records.

For large catalogs, use a staging document type or a generation field. Feed the new generation, validate document counts and sample queries, and then make the new generation available to traffic. This prevents a partial import from replacing a complete index.

Handle changes with webhooks and a queue

WooCommerce webhooks are useful for near-real-time updates. Register events for product creation, product updates, product deletion, and, where the store design requires it, inventory-related changes. The webhook endpoint should acknowledge quickly after authenticating and recording the event. It should not wait for a Vespa feed operation to finish.

Put the event on a durable queue with a deduplication key such as the store ID, object type, object ID, and WooCommerce event ID. A worker can then fetch the current object from WooCommerce before creating the Vespa document. Fetching current state is safer than trusting a webhook payload because events can arrive late, out of order, or with limited product detail.

A practical event flow is:

1. WooCommerce sends a signed webhook request.
2. The integration validates the signature and records the event.
3. The endpoint returns a successful response after durable persistence.
4. A worker coalesces repeated events for the same product or variation.
5. The worker reads the current object from WooCommerce.
6. The transformer creates the complete Vespa projection.
7. The worker feeds, updates, or deletes the deterministic Vespa document.
8. The result, attempt count, and source timestamp are recorded for monitoring.

Make updates idempotent

Webhook systems commonly retry delivery, and queue workers can retry after a timeout even when Vespa accepted the previous request. Reprocessing the same product must therefore produce the same document rather than a duplicate.

Use a stable document ID and send full replacements for normal product synchronization. A full replacement is usually easier to reason about than a sequence of partial field updates because fields removed from WooCommerce are also removed from Vespa. Include the source modification timestamp and reject or defer an older event when a newer version has already been indexed.

For a document type named product in namespace catalog, a feed request can use the Vespa Document API:

curl --request PUT \
  --header "Content-Type: application/json" \
  --data @product.json \
  "https://vespa.example.com/document/v1/catalog/product/docid/store-42-product-184-variation-207"

The exact endpoint and authentication depend on the Vespa deployment. In production, use the Vespa feed client or an equivalent managed worker when you need batching, concurrency control, metrics, and retry behavior.

Delete documents deliberately

A product that is deleted, trashed, made private, or excluded from the catalog must not remain searchable accidentally. Treat non-indexable states as deletion conditions unless the business explicitly needs them for another use case.

For a hard deletion, issue a delete operation for the deterministic document ID:

curl --request DELETE \
  "https://vespa.example.com/document/v1/catalog/product/docid/store-42-product-184-variation-207"

If a parent product has variations, make the deletion logic explicit. Deleting the parent record should either delete all child variation documents or mark them unavailable after querying the integration database for the known child IDs. Do not rely on a search query to discover every document that should be removed.

Separate searchable fields from commerce filters

Search text and transactional filters have different requirements. Title and description fields can use Vespa text indexing and ranking features. Price, stock quantity, product IDs, brand identifiers, and category IDs should use typed attributes or indexes that support efficient filtering and ranking.

For example, a query may combine user text with commerce constraints:

select * from product where 
  ({"targetNumHits":100}userQuery()) and
  stock_status contains "instock" and
  price <= 50

Keep the query logic independent from WooCommerce presentation concerns. The application can use the Vespa result to display search matches, then use WooCommerce or the store’s checkout APIs for authoritative cart and order operations. Never assume that an indexed stock quantity is sufficient to reserve inventory during checkout.

Prevent stale writes

Out-of-order delivery is a common source of stale catalog data. For example, a price update can be processed after a later stock update, and the older product payload can overwrite the newer stock state. There are several ways to reduce this risk:

• Fetch the current WooCommerce object when processing an event instead of applying the event payload directly.
• Store and compare the WooCommerce modification timestamp.
• Use a per-product or per-variation ordering key in the queue.
• Coalesce multiple pending events into one read-and-index operation.
• Use an integration-side version number when the source timestamps are not reliable enough.

Timestamp comparison should account for missing or equal values. When two updates cannot be ordered confidently, prefer a fresh read from WooCommerce and record the ambiguity for monitoring.

Plan for reconciliation

Webhooks are not a complete synchronization strategy. A webhook can be disabled, dropped, rejected because of a timeout, or delivered while the store is being restored from backup. Schedule a reconciliation job for every store.

A lightweight reconciliation can retrieve recently modified WooCommerce products and requeue them. A deeper reconciliation can paginate through the entire catalog, calculate the expected deterministic document IDs, compare them with the integration’s indexed state, and repair missing or stale documents. It should also identify Vespa documents whose source records no longer exist.

Useful operational metrics include webhook acceptance failures, queue age, feed latency, feed errors by store, retry counts, current catalog count, delete count, and the age of the oldest unprocessed source change. Alert on a growing queue and on a mismatch between the WooCommerce catalog count and the expected Vespa document count.

Protect credentials and tenant boundaries

WooCommerce REST credentials and Vespa deployment credentials should be stored in a secret manager, not in product metadata, queue messages, or application logs. Give each store only the permissions it needs. In a multi-tenant Vespa application, include the store identifier in every document and enforce tenant filtering in the query path.

Validate webhook signatures using the WooCommerce webhook secret, reject unexpected content types and oversized requests, and record a correlation ID for every event. Logs should contain store and object identifiers, operation type, latency, and failure details without exposing API keys or customer data.

Test the integration with catalog edge cases

Before deploying the connector for client stores, test more than a simple product update. Include variable products, products with no SKU, duplicate SKUs, scheduled publication, catalog-hidden products, out-of-stock products, category changes, empty descriptions, special characters, deleted variations, restored products, rapid successive price changes, webhook retries, and a failed Vespa feed followed by a retry.

For each case, verify both the document content and search behavior. A reliable integration is one where a product can be changed repeatedly in WooCommerce, events can be delivered out of order, workers can restart, and the final Vespa projection still converges to the current source state.