Install Weaviate documentation with WPSOLR

Videos tutorials

Coming soon…

This guide shows how to install the Weaviate vector database using Docker compose with any vectorizing model you want. It will explain how to setup SSL/TLS protection (by setting it up with Weaviate options in the Docker compose file or reverse proxy with nginx), setup authentification and permissions, manage clusters and setup replication, create collections (classes) and send basic queries.

Table of Contents

Startup a simple Weaviate container

You can download a docker-compose file with your specific configuration you want from the official Weaviate website or just copy the content from down below (uses the paraphrase-multilingual-MiniLM-L12-v2 sentence transformers module, multilingual, 384 dimensional vectors) and add it to a docker-compose.yml file :

version: '3.4'
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.21.3
    ports:
    - 8080:8080
    volumes:
    - ./weaviate_data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      TRANSFORMERS_INFERENCE_API: 'https://t2v-transformers:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
      ENABLE_MODULES: 'text2vec-transformers'
      CLUSTER_HOSTNAME: 'node1'
      
    networks:
      - weaviate_net

  t2v-transformers:
    image: semitechnologies/transformers-inference:sentence-transformers-paraphrase-multilingual-MiniLM-L12-v2
    environment:
       ENABLE_CUDA: '0'

networks:
  weaviate_net:

Start the container with the following bash command:

docker-compose up -d

Access it using the following command :

curl https://example.org:8080/

Use any Huggingface model you want (Optional)

We also want to be able to use any model as a Wearviate vectorizer. If you want to find the right one for you you can use the huggingface rankings page. I chose the “BAAI/bge-small-en-v1.5” model for this guide because it’s the smallest better performer.

According to the t2v-transformers-models page on github, we need to create a custom inference container containing our custom model to be used by weaviate.

To create this container, you need to create this Dockerfile named “Dockerfile_custom_model” with the following content :

FROM semitechnologies/transformers-inference:custom
RUN MODEL_NAME=BAAI/bge-small-en-v1.5 ./download.py

You can use any model from huggingface you want instead of “BAAI/bge-small-en-v1.5” as the “MODEL_NAME” value (For example : sentence-transformers/all-MiniLM-L6-v2, etc…).

You can then create a Dockerfile with this content :

version: '3.4'
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.21.3
    ports:
    - 8080:8080
    volumes:
    - ./weaviate_data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      TRANSFORMERS_INFERENCE_API: 'https://t2v-transformers:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
      ENABLE_MODULES: 'text2vec-transformers'
      CLUSTER_HOSTNAME: 'node1'
  t2v-transformers:
    build:
      context: .
      dockerfile: Dockerfile_custom_model
    image: weaviate_custom_model
    environment:
      ENABLE_CUDA: '0'

A separate container named “t2v-transformers” containing our model (by building the image from the previously created “Dockerfile_custom_model”) has now been created.

If you want to use the GPU, you can set as the environment variable :

      ENABLE_CUDA: '1'

You can now start it using :

docker-compose up -d

Once again, access it using the following command :

curl https://example.org:8080/

Setup authentification and permissions

Add authentification

Add these lines to the environment variables section of the weaviate service in the docker-compose.yml file :

AUTHENTICATION_APIKEY_ENABLED: 'true'
AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'admin-secret-key,user1-secret-key, user2-secret-key'
AUTHENTICATION_APIKEY_USERS: '[email protected], [email protected], [email protected]'

The AUTHENTICATION_APIKEY_ALLOWED_KEYS line defines the secret key that you enter in each request to authentificate and in the same position of the

AUTHENTICATION_APIKEY_USERS line you add the corresponding email address.

Define permissions

Same thing, add these lines to the environment variables section of the weaviate service in the docker-compose.yml file :

AUTHORIZATION_ADMINLIST_ENABLED: 'true'
AUTHORIZATION_ADMINLIST_USERS: '[email protected]'
AUTHORIZATION_ADMINLIST_READONLY_USERS: '[email protected], [email protected]'

Enable permissions control with the AUTHORIZATION_ADMINLIST_ENABLED line, create admins by adding the users’ email adresses to the AUTHORIZATION_ADMINLIST_USERS environment variable and finally add readonly users by adding the users’ email adresses to the AUTHORIZATION_ADMINLIST_READONLY_USERS environment variable.

Admins can do anything (modify the server config, add data, send queries) while readonly users can only send queries.

You can send any request with the Authorization header containing your secret key like the following :

curl https://localhost:8080/v1/nodes  -H "Authorization: Bearer admin-secret-key"

Configure SSL/TLS protection (Optional)

You first need to generate the certificate

You can generate a keypair either by using a certificate authority service like Let’s Encrypt, or generate them yourself using the following :

mkdir weaviate_ssl_certs
cd weaviate_ssl_certs
openssl genrsa -out cert.key 2048
openssl req -new -x509 -sha256 -key cert.key -subj /CN=example.org -out cert.pem -days 730

By adding options to the docker compose file

In the docker-compose.yml file, add lines to the command section of the weaviate service :

  weaviate:
    command:
      #- --host
      #- 0.0.0.0
      #- --port
      #- '8080'
      - --tls-host
      - 0.0.0.0
      - --tls-port
      - '8080'
      - --scheme
      #- http
      - https
      - --tls-certificate
      - /etc/ssl/certs/cert.pem
      - --tls-key
      - /etc/ssl/certs/cert.key
    image: semitechnologies/weaviate:1.21.3
    ports:
      - 8080:8080
    volumes:
      - ./weaviate_data:/var/lib/weaviate
      - ./weaviate_ssl_certs/:/etc/ssl/certs/

Mount the folder containing your certificate & private key pair into the weaviate container, and change the scheme to https and define the path to the key pair.

You can then send request to the container using the following :

curl https://example.org/v1/nodes  -H "Authorization: Bearer admin-secret-key"

or if you have a self-signed certificate :

curl --cacert weaviate_ssl_certs/ca.pem https://example.org

Since it is a self-signed certificate and inherently insecure, you need to add the –cacert option followed by the path to the certificate to every query to verify the server yourself.

By setting up a reverse proxy using NGINX

Create a nginx.conf file, and add the following content in it :

upstream weaviate{
    server weaviate:8080;
}

server {
        # You would want to make a separate file with its own server block for each virtual domain
        # on your server and then include them.
    listen       80;
        #tells Nginx the hostname and the TCP port where it should listen for HTTP connections.
        # listen 80; is equivalent to listen *:80;
    server_name  example.org;

    location / {
        #The location setting lets you configure how nginx responds to requests for resources within the server.
        return 301 https://localhost$request_uri;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }

}
    
# HTTPS server
#
server {
   listen       443 ssl;
   server_name  example.org;

   ssl_certificate      /etc/nginx/ssl/localhost/ca.pem;
   ssl_certificate_key  /etc/nginx/ssl/localhost/ca-key.pem;

   location / {
        proxy_pass https://weaviate;
   }

}

In the upstream section, add the url of the weaviate container relative to the NGINX container. In this case, since Weaviate is in the same docker network, it’s url is the name of the service.

The NGINX listens on the port 80 but redirects the user to the https on port 443.

In the HTTPS server section, the server name is the url of NGINX relative to the client. In this case, it is example.org.

Add the NGINX service to the docker-compose.yml file :

  nginx:
    image: nginx:latest
    ports:
    - 80:80
    - 443:443
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf
      - ./weaviate_ssl_certs/:/etc/nginx/ssl/localhost/
    networks:
      - weaviate_net

Start the services with :

docker-compose up -d

You can then connect to the Weaviate using :

curl -L https://example.org/v1/nodes  -H "Authorization: Bearer admin-secret-key"

or if you are using self-signed certificates :

curl -L --cacert weaviate_ssl_certs/ca.pem https://localhost/v1/nodes  -H "Authorization: Bearer admin-secret-key"

The -L option will ensure that it will redirect to https in case an http address is entered.

If you ever need to apply changes to the NGINX configuration, restart the container using the following :

docker restart nginx_container

Setup a cluster (Optional)

If you have multiple nodes, you can create a Weaviate cluster.

To set it up, you can first add another similar weaviate container in the docker-compose.yml file like the following :

  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.21.3
    ports:
    - 8080:8080
    - 3456:3456
    volumes:
    - ./weaviate_data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      TRANSFORMERS_INFERENCE_API: 'https://t2v-transformers:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
      ENABLE_MODULES: 'text2vec-transformers'
      AUTHENTICATION_APIKEY_ENABLED: 'true'
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'admin-secret-key,user1-secret-key, user2-secret-key'
      AUTHENTICATION_APIKEY_USERS: '[email protected], [email protected], [email protected]'
      AUTHORIZATION_ADMINLIST_ENABLED: 'true'
      AUTHORIZATION_ADMINLIST_USERS: '[email protected]'
      AUTHORIZATION_ADMINLIST_READONLY_USERS: '[email protected], [email protected]'
      CLUSTER_HOSTNAME: 'node1'
     networks:
       - weaviate_net

  weaviate2:
    command:
      - --host
      - 0.0.0.0
      - --port
      - '8080'
      - --scheme
      - http
    image: semitechnologies/weaviate:1.21.3
    ports:
      - 8081:8080
      - 3457:3457
    volumes:
      - ./weaviate_data2:/var/lib/weaviate
      - ./weaviate_ssl_certs/weaviate/:/etc/ssl/certs/
    restart: on-failure:0
    environment:
      TRANSFORMERS_INFERENCE_API: 'https://t2v-transformers:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
      ENABLE_MODULES: 'text2vec-transformers'
      AUTHENTICATION_APIKEY_ENABLED: 'true'
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'admin-secret-key,user1-secret-key, user2-secret-key'
      AUTHENTICATION_APIKEY_USERS: '[email protected], [email protected], [email protected]'
      AUTHORIZATION_ADMINLIST_ENABLED: 'true'
      AUTHORIZATION_ADMINLIST_USERS: '[email protected]'
      AUTHORIZATION_ADMINLIST_READONLY_USERS: '[email protected], [email protected]'
      CLUSTER_HOSTNAME: 'node2'
    networks:
      - weaviate_net

You can the add the following environment variables to the weaviate service in the docker-compose.yml:

      CLUSTER_HOSTNAME: 'node1'
      CLUSTER_GOSSIP_BIND_PORT: '7100'
      CLUSTER_DATA_BIND_PORT: '7101'
      CLUSTER_BASIC_AUTH_USERNAME: admin
      CLUSTER_BASIC_AUTH_PASSWORD: password

This is the founder node. Other nodes will join it.

Then you can do the same to the weaviate2 service :

      CLUSTER_HOSTNAME: 'node2'
      CLUSTER_GOSSIP_BIND_PORT: '7100'
      CLUSTER_DATA_BIND_PORT: '7101'
      CLUSTER_JOIN: 'weaviate:7100'
      CLUSTER_BASIC_AUTH_USERNAME: admin
      CLUSTER_BASIC_AUTH_PASSWORD: password

This node will join the first weaviate container.

To check the cluster state :

curl https://example.org:8080/v1/nodes  -H "Authorization: Bearer admin-secret-key"

If you had set up ssl/tls with a reverse proxy using NGINX, you should add the new node to the upstream section in the nginx.conf like the following :

upstream weaviate{
    server weaviate:8080;
    server weaviate2:8080;
}

Don’t forget to set the load balancing method as well in this section. By default, it is round-robin , but you can check out all the different methods in the official documentation.

Import the data

You can connect to your weaviate server using the Weaviate python client :

pip install weaviate-client

import weaviate

client = weaviate.Client(
    url="https://localhost:8080",
    auth_client_secret=weaviate.AuthApiKey(api_key="admin-secret-key")
)

We can then create the Class. A class is a collection of the elements you index. If you have properties associated to the vectors you import you can define them in the class. Each property is defined by a name and a datatype.

class_obj = {
    "class": "Sentences",
    "vectorizer": "none",
    "properties": [
        {
            "name": "passage",
            "dataType": ["text[]"]
        },
        {
            "name": "answer",
            "dataType": ["text[]"]
        },
        {
            "name": "query",
            "dataType": ["text"]
        },
        ],
        'vectorizer': 'text2vec-transformers'
    }


client.schema.create_class(class_obj)

Or if you want to enable replication for this class :

class_obj = {
    "class": "Sentences",
    "vectorizer": "none",
    "properties": [
        {
            "name": "passage",
            "dataType": ["text[]"]
        },
        {
            "name": "answer",
            "dataType": ["text[]"]
        },
        {
            "name": "query",
            "dataType": ["text"]
        },
        ],
        'vectorizer': 'text2vec-transformers',
        'replicationConfig': {
          'factor': 2
        }
    }

If you need to, you can delete the class using :

client.schema.delete_class("Sentences")

The properties of the object will be displayed to the user when sending a query and finding a match. For this guide, the data will come from the ms_marco dataset but you can use any type of data you want to. The ms_marco dataset has “answer”, “query” and “passage” columns so if you use your own data you can have whatever property and how many or little you need.

In this step, you can create the embeddings and add them as well as the properties to the elements array.

from datasets import load_dataset

# Import the "ms_marco" dataset and load the sentences
dataset = load_dataset("ms_marco", 'v1.1')
passages_data = dataset["train"]['passages']
answers_data = dataset["train"]['answers']
query_data = dataset["train"]['query']

elements = []

# Select the 50 first sentences of the dataset
for i in range(50):
    element = {}
    passage = passages_data[i]['passage_text']
    answer = answers_data[i]
    query = query_data[i]

    # Create the respective embedding
    element["Passage"] = passage
    element["Answer"] = answer
    element["Query"] = query

    elements.append(element)

You can the Weaviate batch function to add all the elements of the elements array to the “Sentences” class. This will send the elements to Weaviate 10 at a time.

client.batch.configure(batch_size=10)  # Create a batch of 10

with client.batch as batch:
    # Batch import all Questions
    for i, d in enumerate(elements):
        properties = {
            "passage": d["Passage"],
            "answer": d["Answer"],
            "query": d["Query"],
        }

        batch.add_data_object(properties, "Sentences")

The data sent to the Weaviate container will automatically be vectorized by our Huggingface model. If we didn’t use a vectorization mode, we could add an optional vector parameter to add_data_object method (you can check the documentation for more).

Now that you have added the vectors, you can send queries to the Weaviate database. In this case “What lives in water”.

import json

query = "What to do with food"

nearText = {
  "concepts": [query],
  "distance": 0.6
  }

result = client.query.get(
        "Sentences", ["passage", "answer", "query"]
    ).with_near_text(
        nearText
    ).with_limit(2).with_additional(['certainty']).do()

print(json.dumps(result, indent=4))

If you want to check that a vector was correctly created you can add a ‘vector’ element to the array in the “with_additional” method.