How to Import Millions of Products Into Algolia

Importing a few thousand WooCommerce products into Algolia is straightforward. Importing several million products, variations, inventory records, and taxonomy combinations requires a controlled indexing pipeline. The important design decisions are record shape, batching, concurrency, retries, incremental updates, and how you switch from an old index to a newly built one.

Define the Algolia record before building the importer

Do not send the entire WooCommerce product object to Algolia. Search records should contain the fields required for retrieval, filtering, sorting, merchandising, and display. Keep large descriptions, internal metadata, and fields that are never queried out of the index.

A product record might look like this:

{
  "objectID": "product_48291",
  "type": "product",
  "product_id": 48291,
  "sku": "SHOE-48291",
  "name": "Trail Running Shoe",
  "slug": "trail-running-shoe",
  "brand": "Example Brand",
  "categories": ["Men", "Shoes", "Running"],
  "category_ids": ["12", "44", "91"],
  "price": 129.99,
  "sale_price": 109.99,
  "in_stock": true,
  "stock_quantity": 18,
  "image_url": "https://store.example.com/images/trail-running-shoe.jpg",
  "url": "https://store.example.com/product/trail-running-shoe/"
}

Use a deterministic objectID. A WooCommerce product ID is suitable when each product is one searchable record. If variations are independently searchable, use a namespaced identifier such as variation_873421. Never generate a random ID during an import, because rerunning the job would create duplicates instead of replacing existing records.

Choose the product-to-record model

Most WooCommerce stores use one of three models:

• One record per parent product: best when shoppers search for products and select size, color, or other options on the product page.
• One record per variation: useful when variations have different prices, inventory, images, or searchable attributes.
• One product record with nested variants: useful when the search result should represent one product while filtering must account for variant data.

Make this decision before importing millions of records. Changing from parent products to variations changes object IDs, facet fields, record counts, and the logic required for updates and deletes.

Extract WooCommerce data outside the web request

A full import should not run inside a WordPress request, an administrator page load, or a single WP-Cron callback. Those execution models are vulnerable to PHP timeouts, memory limits, reverse-proxy timeouts, and deployment interruptions.

Use a dedicated worker or command-line process that reads products in pages. For large catalogs, prefer a purpose-built export query or a queue-backed data extraction process over repeatedly loading every product through expensive WordPress object APIs. The exact implementation depends on the store’s extensions and data model, but the extraction layer should:

• Read products and variations in stable pages, using IDs or another indexed cursor.
• Resolve only the taxonomies, prices, stock values, media, and metadata needed by the record schema.
• Normalize prices as numbers rather than formatted currency strings.
• Convert WooCommerce visibility, catalog visibility, stock, and publication rules into explicit indexable states.
• Exclude products that should never appear in search.
• Record the source version, last-modified timestamp, or content hash for each exported item.

Offset pagination can become increasingly expensive on very large tables. A cursor such as WHERE ID > last_seen_id ORDER BY ID, supported by appropriate indexes and a stable source query, is usually more reliable for a full export.

Send records in batches

Algolia’s indexing APIs are designed for batched writes. Do not send one HTTP request per product. Build a batch of records, send it with the official server-side client, and persist the import position only after the batch has been accepted.

Batch size should respect the current Algolia API limits and the serialized request size. A batch of 500 or 1,000 compact records may work well, but the correct value depends on record size and the endpoint being used. Measure payload sizes rather than assuming that a fixed number of products is always safe.

import algoliasearch from 'algoliasearch';

const client = algoliasearch(
  process.env.ALGOLIA_APPLICATION_ID,
  process.env.ALGOLIA_ADMIN_API_KEY
);

const index = client.initIndex('products_tmp_20250308');

async function sendBatch(records) {
  if (records.length === 0) return;

  const response = await index.saveObjects(records, {
    autoGenerateObjectIDIfNotExist: false
  });

  return response;
}

Use an Algolia Admin API key only in the worker or another trusted server environment. It must never be exposed in browser JavaScript, a WordPress page, a public repository, or a client-side configuration file.

Control concurrency instead of opening unlimited requests

Millions of records require throughput, but unrestricted parallel requests can cause rate limiting, connection exhaustion, and difficult recovery. Use a bounded worker pool. For example, a queue can process four to eight batches concurrently, then reduce concurrency when it receives throttling responses.

Implement retries for transient failures such as network errors and rate limits. Use exponential backoff with jitter, and honor the retry information returned by the API when available. Do not retry malformed records indefinitely. Send those records to a dead-letter queue containing the product ID, validation error, and source payload or export reference.

Indexing operations are asynchronous. The client returns task information that can be used to monitor progress. Waiting synchronously for every task before sending the next batch can severely reduce throughput. A high-volume importer should track accepted tasks, poll them in a separate monitoring process when confirmation is required, and define how many outstanding tasks are safe for the application and plan.

Build a temporary index for a full rebuild

For a major schema change or a complete reimport, write to a new index such as products_tmp_20250308 instead of deleting the live index first. Apply the production settings, searchable attributes, facets, ranking configuration, synonyms, and replicas to the temporary index before importing.

After validation, switch the application to the rebuilt index. For applications that require an atomic index replacement, use Algolia’s index move or replace operation according to the client and API version in use. Validate how replicas and index settings are handled in the selected operation before using it in production. Keep the previous index until search traffic and business checks confirm that the new data is correct, then remove it deliberately.

Do not treat an index alias or an application-side configuration change as a substitute for validating replica names, dashboard configuration, analytics targets, and any code that constructs index names dynamically.

Keep the catalog current with delta updates

A full import establishes the baseline; it should not be the normal way to process every product change. WooCommerce stores should emit update events when a product, variation, price, stock quantity, taxonomy relationship, visibility value, or relevant media field changes.

Place the product ID on a durable queue and coalesce repeated events. If price and stock change ten times during a bulk operation, one final reindex is often enough. The worker should rebuild the complete Algolia record from the current source of truth rather than applying a fragile sequence of partial assumptions.

For deletions, enqueue the stable objectID and call the delete operation. A product that becomes private, out of stock, or excluded from the catalog may need an update rather than a delete, depending on whether the storefront supports backorder, hidden inventory, or alternate search rules.

Prevent stale workers from overwriting newer data

Large imports and live updates often run at the same time. A slow full-import worker can otherwise write an old product version after a newer stock or price update has already been indexed.

Use a source revision or modified timestamp in the queue message. Before writing, compare the message revision with the current source revision, or route all writes through a version-aware indexing service. At minimum, run a reconciliation job after the full import that finds records changed during the import window and reindexes them from current WooCommerce data.

Configure settings before measuring search quality

Importing records does not automatically produce useful search behavior. Configure and test:

• searchableAttributes for product names, SKUs, brands, and relevant keywords.
• attributesForFaceting for brand, category IDs, stock state, price ranges, and other filterable values.
• customRanking for commercial signals such as sales, margin, popularity, or inventory policy.
• Replica indexes for price sorting, newest products, or other alternate ranking requirements.
• Synonyms, typo tolerance, query rules, and merchandising rules.

Keep facet values compact and consistent. For example, use stable category IDs for filtering and display category labels separately. Avoid indexing an unrestricted set of internal metadata keys as facets, because high-cardinality attributes increase index size and make the search configuration harder to govern.

Validate a multi-million-record import

Count comparisons are necessary but not sufficient. Before switching production traffic, validate representative records and business behavior:

1. Compare the source export count with the number of accepted records and the number of rejected records.
2. Check that every record has a unique, stable objectID.
3. Sample published, variable, discounted, out-of-stock, and excluded products.
4. Verify prices, currency interpretation, URLs, images, categories, and variation attributes.
5. Run representative keyword, SKU, category, facet, and sort queries.
6. Confirm that records intentionally removed from WooCommerce are absent from the target index.
7. Measure indexing throughput, task lag, error rates, record size, and worker queue depth.

A reliable importer is therefore a resumable data pipeline rather than a one-time script: extract in stable pages, transform into a deliberately small record, batch writes within API limits, control concurrency, retry safely, isolate failures, rebuild through a temporary index when appropriate, and process subsequent WooCommerce changes through a durable delta queue.