1. Google PaLM vectorizer

Google PaLM vectorizer uses the Google PaLM API to create Embeddings (vectors) from data. Vectorizing from large language models is CPU intensive, often requires GPUs, and can be very ineffective without proper optimisations. This is the purpose of this module, which calls Google PaLM servers with their Embeddings API.

 

Create a docker compose file from Weaviate wizzard : select the Text transformers vectorizer, with a specific transformer model among the list. Then start the docker container.

• (1) Select the “Text” vectorizer type
• (2) Select the “Google PaLM” vectorizer
• (3) Leave it. The API key will be set later, in WPSOLR’s settings.

Download and execute the docker-compose file generated by the wizard (docker-compose up -d):

 

Create a GCP project with Vertex AI authorization

Before starting, check out the Google PaLM Embedding models pricing for the model “textembedding-gecko-001”.

 

1. Select or create your Google Cloud project
2. Click on the APIs menu
3. Click on the button “Enable APIs”
4. Select and activate the “Vertex AI” API

 

1. Select project menu
2. Select menu “IAM & Admin
3. Select menu “Service Accounts”

 

1. Click on “+ CREATE SERVICE ACCOUNT”

 

1. Enter the new service account name
2. Enter the new service account details
3. Click on “Create and continue”

 

1. Click on “Role”
2. Select your role
3. Click on “Done”

 

 

1. Click on the menu of the new service account
2. Select “Manage keys”

 

1. Click on tab “Keys”
2. Click on “Add key”
3. Select “json”
4. Click on “create”

 

1. The json private key is downloaded on your computer

 

1. Open your key.json and copy its content. It will be used later during the WPSOLR setup.

 

Now, let’s create our index in WPSOLR:

• (1) Select the text2vec-palm vectorizer module
• (2) Set a name for you index, visible in WPSOLR admin
• (3) Set a name for your Weaviate class (index)
• (4) Set the url of your Weaviate docker container
• (5) Copy the content of your Google Cloud JSON key described earlier
• (6) Set the PaLM model
• (7) Create the index. Done!

 

Connect to the Weaviate GraphQL console at https://console.semi.technology/console with url https://localhost:8080, to check our new index (class):

2 Select your data

• (1) (2) (3) select the index you just created
• (4) Choose a filter: “Near Text” to perform a vector search (search on concepts), or “Where” to perform a keywords search (classic search that works with words)
• (5) Set a similarity for your “Near Text” search. The closer to “1”, the more precise is the vector search.

 

3 Index your data