A reliable content ingestion pipeline turns WordPress and WooCommerce data into consistent, searchable documents. For an agency, this is more than moving posts into an index: the pipeline must handle product updates, variation changes, custom fields, multilingual content, deletions, retries, and temporary failures without creating stale or duplicate search results.
Define the ingestion contract first
Before connecting WordPress to WPSOLR or another search service, define the document contract that every indexed record must follow. A useful contract specifies the source identifier, document type, title, body, URL, taxonomies, product attributes, visibility state, language, modification timestamp, and any AI-generated fields.
{
"source_id": "product:4821",
"post_id": 4821,
"type": "product",
"title": "Merino Hiking Socks",
"url": "https://example.com/product/merino-hiking-socks/",
"status": "publish",
"modified_gmt": "2025-02-14T10:30:00Z",
"taxonomies": {
"product_cat": ["Hiking", "Accessories"]
},
"attributes": {
"pa_material": ["Merino wool"]
}
}
Use a stable identifier such as product:4821, rather than a title or URL. Titles and URLs can change; a stable identifier allows updates to replace the correct document and makes retries idempotent.
Separate extraction, transformation, and loading
Keep the pipeline in three explicit stages:
- Extract: Read the WordPress or WooCommerce object, including its status, metadata, taxonomies, relationships, and modification date.
- Transform: Normalize the object into the ingestion contract. Remove presentation-only markup, resolve selected custom fields, and prepare content for search or AI processing.
- Load: Send the normalized document to the WPSOLR indexing workflow or queue it for asynchronous processing.
This separation makes the system easier to test. A transformation test can run against a fixture without connecting to a live index, while loading tests can verify retry and failure behavior using a mock endpoint.
Use events for freshness and scheduled reconciliation for completeness
Hooks and webhooks provide fast updates. For example, a product save, stock change, category update, or post transition can enqueue the affected object. However, event-driven ingestion alone is not sufficient. Bulk imports, failed requests, plugin conflicts, direct database changes, and missed cron jobs can leave the index incomplete.
Use two complementary mechanisms:
- Incremental ingestion: Enqueue individual objects when relevant WordPress or WooCommerce events occur.
- Reconciliation: Run a scheduled job that compares recently modified records, verifies deletion markers, and optionally samples indexed documents against their WordPress source.
A reconciliation job should be bounded. Process a fixed number of records per run, store a cursor, and avoid a single request that attempts to rebuild an entire catalog.
Make queue jobs idempotent
Queues normally provide at-least-once delivery. The same job may run more than once after a timeout or worker restart, so processing must be safe to repeat. A job should carry the object type, object ID, source modification timestamp, operation, and an attempt count.
class Content_Ingestion_Job {<br> public function handle( $object_type, $object_id, $modified_gmt, $operation ) {<br> $current = $this->load_source_object( $object_type, $object_id );<br><br> if ( 'delete' === $operation || ! $current || 'publish' !== $current->post_status ) {<br> return $this->delete_from_index( $object_type, $object_id );<br> }<br><br> if ( $current->post_modified_gmt !== $modified_gmt ) {<br> return $this->enqueue_latest_version( $object_type, $object_id, $current->post_modified_gmt );<br> }<br><br> $document = $this->transform( $current );<br> return $this->upsert( $document );<br> }<br>}
The worker should re-read the current source object rather than trusting a stale event payload. If the object changed after the job was created, enqueue the latest version instead of indexing an older snapshot.
Handle WooCommerce-specific changes
WooCommerce data is distributed across products, variations, taxonomies, attributes, prices, stock metadata, and visibility settings. Decide whether variations should be indexed as separate documents, embedded in the parent product, or handled using both approaches.
Separate variation documents are useful when customers search by size, color, SKU, or availability. Parent-only documents are simpler, but they must include enough variation data to support filtering. Whichever model you select, document it in the ingestion contract and use deterministic identifiers such as product_variation:9132.
Be especially careful with:
- Product status and catalog visibility.
- Stock status and backorder settings.
- Price and sale price changes.
- Global attributes and custom product attributes.
- Parent-child relationships for variations.
- Product category and tag changes that affect many documents.
- Deletion, trash, and permanent deletion events.
A taxonomy change may require reindexing every related product, not only the term object. For large catalogs, enqueue the affected product IDs in batches and rate-limit the workers.
Normalize content before AI enrichment
AI processing should consume a controlled representation of the source content. Strip navigation, shortcodes that have no search value, tracking parameters, and duplicated template text. Preserve meaningful headings, product specifications, FAQs, and editorial content.
Do not overwrite the source content with generated output. Store enrichment separately, identify the model and prompt version, and retain the source modification timestamp used to create it. For example, an indexed document might contain:
content_plainfor keyword search.content_chunksfor chunk-level retrieval.ai_summaryfor a concise generated summary.ai_keywordsfor controlled enrichment.embedding_versionfor tracking vector compatibility.
When the source changes, invalidate derived fields created from the previous version. A content hash is useful for detecting whether an expensive enrichment step is actually necessary.
$content_hash = hash( 'sha256', wp_json_encode( array(<br> 'title' => $document['title'],<br> 'content' => $document['content_plain'],<br> 'fields' => $document['custom_fields'],<br>) ) );
Design retries around failure types
Not every failure should be retried. Network timeouts, temporary rate limits, and service-unavailable responses are usually transient. Invalid field mappings, malformed documents, authentication failures, and unsupported content types require an operator or deployment change.
Use exponential backoff with jitter for transient failures. Cap the number of attempts, then move the job to a dead-letter queue containing the source ID, payload reference, error class, and last response. Avoid storing sensitive customer data in logs or dead-letter records.
When a batch request partially fails, retry only the failed documents if the destination provides per-document results. Retrying an entire batch can amplify traffic and make a single malformed record obscure successful work.
Prevent stale writes and race conditions
Multiple workers can process different versions of the same object out of order. Include the source modification timestamp or a monotonically increasing version in the document. The loader should reject an older version when a newer version has already been accepted.
For high-value catalogs, maintain an ingestion ledger with the source ID, last processed version, operation, result, and timestamp. This ledger provides an audit trail and supports targeted replay without rebuilding the entire index.
Make deletion a first-class operation
Unpublishing or deleting a post is not the same as receiving no update. If the source disappears before the worker runs, the worker still needs enough information to remove the indexed document. Include the source type and ID in deletion events, and retain a short-lived tombstone or deletion record when necessary.
Scheduled reconciliation should also identify documents that exist in the index but no longer exist in WordPress. This is particularly important after migrations, bulk deletions, slug changes, and product catalog imports.
Measure the pipeline
Track operational metrics that expose freshness and data quality:
- Queue depth and oldest pending job age.
- Successful, retried, and permanently failed jobs.
- Ingestion latency from source modification to indexed version.
- Documents indexed, updated, and deleted per object type.
- Transformation failures by plugin, field, or content type.
- Reconciliation discrepancies.
- AI enrichment cost, duration, and cache-hit rate.
Expose a small administrative report for agency teams. It should show the last successful run, current backlog, recent failures, and a way to replay a specific source ID. For client support, log correlation IDs so a product update can be traced from the WordPress event through the queue and into the WPSOLR indexing response.
Test with production-shaped fixtures
Unit tests should cover published and unpublished posts, products with variations, missing metadata, multilingual fields, private content, malformed shortcodes, and deletion events. Integration tests should verify that an update replaces the existing document rather than creating a duplicate.
Before launch, run a dry-run import that reports the documents and fields that would be sent without changing the index. Compare counts by post type, language, catalog visibility, and taxonomy. Then perform a controlled backfill with a bounded worker pool, monitor failures, and run reconciliation after the initial load.