Turn WordPress content and documents into AI-ready knowledge—without burdening WordPress
Organizations store valuable knowledge across WordPress custom post types, media libraries, ACF file fields, and external documents hosted on Google Drive. Making this information available to an internal AI assistant requires a pipeline that can process large volumes of documents without slowing down the website.
In this use case, WPSOLR supports the content ingestion and indexing workflow. The client’s development team built the internal chatbot, using Weaviate as its vector database for retrieval.
1. Collect WordPress content and document links
The pipeline collects selected WordPress post types and their associated ACF file fields, including links to media library files and Google Drive documents.
WordPress media files are stored in Google Cloud Storage, providing a central location for the document-processing pipeline.
2. Process documents asynchronously
WordPress sends post data and embedded file references to a Google Cloud Pub/Sub queue. A custom Python application running in Docker consumes queued messages and processes the associated content outside the WordPress application.
The pipeline handles PDFs, Office documents, video, and audio. It extracts their content, splits it into chunks, generates embeddings where required, and prepares the chunks for indexing.
3. Scale processing capacity automatically
The architecture is designed to handle fluctuating workloads. When the Pub/Sub queue fills up during periods of heavy ingestion, the processing infrastructure can scale to up to 100 Docker containers to process queued documents in parallel.
Instead of making WordPress handle resource-intensive document extraction and chunking directly, the workload is offloaded to independent workers. This keeps the ingestion process asynchronous and minimizes the performance impact on the WordPress website.
4. Index chunks in Weaviate
The processed document chunks are indexed in Weaviate, which serves as the vector database for semantic retrieval. The chatbot can then retrieve relevant passages from the indexed content rather than relying only on document titles or links.
5. The client builds the chatbot
The client’s development team built and owns the internal chatbot, using Weaviate to retrieve relevant document content and support employees’ natural-language questions.
WPSOLR supports the content ingestion and indexing pipeline, while the client controls the chatbot application and conversational experience.
The result
- Scalable processing: Up to 100 Docker containers can handle heavy document-ingestion workloads in parallel.
- Minimal WordPress performance impact: Resource-intensive processing runs outside WordPress.
- Multiple file formats: PDFs, Office documents, video, and audio are processed into searchable chunks.
- Semantic retrieval: Weaviate indexes the chunks for the client’s chatbot.
- Asynchronous architecture: Google Cloud Pub/Sub separates content collection from document processing.
The takeaway: A scalable, cloud-based document pipeline turns WordPress content and linked files into AI-ready knowledge, while keeping the website responsive and giving the client full control over its internal chatbot.