Comparison of vector search vs. traditional keyword search

Introduction

Search engines are essential tools to help users find information on websites. Traditional keyword searches are commonly used, but they may not always provide the best search results. As a solution, vector search was introduced. This type of search has several advantages over traditional keyword searches.

In this article, we will compare vector search to traditional keyword search, and we will discuss how WPSOLR can help improve search functionality on WordPress websites. Additionally, we will provide sample code with a PHP client, embedded in HTML tags.

Comparison of Vector Search vs Traditional keyword search

Keyword search is commonly used and relatively easy to implement. This type of search involves matching the keywords in a search query to the keywords in a database. Traditional keyword searches use an exact match algorithm, meaning that the search results will only show documents containing the exact keyword(s) being searched for.

On the other hand, vector search is more complex. It operates by first creating a vector space model, which represents each document as a vector of numerical values. These values correspond to the frequencies of each word in the document. After creating the vector space model, vector search finds the cosine similarity between the query and each document’s vector. Finally, it ranks the documents based on the cosine similarity and returns the results.

Vector search has several advantages over traditional keyword searches. It can provide more relevant search results, as it takes into account the context of the words in the documents. Vector search can also handle synonymy and polysemy better than traditional keyword searches. Synonymy is when two words have the same meaning, and polysemy is when a word has multiple meanings. Vector search can map these variations to a semantic space and retrieve all relevant documents, regardless of the exact keyword used in the search query.

PHP Client example

// Example of a simple function to search documents using vector search

function search($query) {
    // Connect to the database
    $db = mysqli_connect('localhost', 'user', 'password', 'database');

    // Tokenize the query
    $query_tokens = explode(' ', $query);

    // Create the query vector
    $query_vector = array();
    foreach($query_tokens as $token) {
        $query_vector[$token]++;
    }

    // Perform the vector search
    $results = array();
    $documents = mysqli_query($db, 'SELECT * FROM documents');
    while($document = mysqli_fetch_assoc($documents)) {
        // Create the document vector
        $document_vector = array();
        $document_tokens = explode(' ', $document['content']);
        foreach($document_tokens as $token) {
            $document_vector[$token]++;
        }

        // Calculate the cosine similarity
        $similarity = 0;
        foreach($query_vector as $term => $frequency) {
            if(array_key_exists($term, $document_vector)) {
                $similarity += $frequency * $document_vector[$term];
            }
        }
        $similarity /= sqrt(array_sum(array_map(function($x) { return pow($x, 2); }, array_values($query_vector)))) * sqrt(array_sum(array_map(function($x) { return pow($x, 2); }, array_values($document_vector))));
        $results[$document['id']] = $similarity;
    }

    // Sort the results and return the document IDs
    arsort($results);
    $ids = array_keys($results);
    return $ids;
}

// Example usage
$ids = search('vector search');
print_r($ids);

How WPSOLR Can Help

WPSOLR is a plugin for WordPress that can enhance search functionality on any WordPress website. This plugin provides several additional features, including vector search. With WPSOLR, it is possible to create an index of all website content, including posts, pages, comments, and custom post types. This index can then be searched using vector search to provide more relevant search results and improve the user experience.
In conclusion, vector search has several benefits compared to traditional keyword searches. It provides more relevant search results, handles synonymy, and polysemy better and considers the context of words in documents before ranking them. Implementing vector search may be challenging, but with the help of plugins such as WPSOLR, it has become more accessible and easier to implement.