Milvus

Milvus Documentation

This guide provides detailed instructions on installation, configuring, implementing, and troubleshooting milvus, a vector database for similarity search and AI applications. It covers all necessary steps with relevant screenshots and examples to help students understand the usage of milvus for the IT Support Chatbot project.

What is Milvus?

Milvus is an open-source vector database optimized for handling and searching large-scale vector data, commonly used in AI applications like recommendation systems and similarity search. It supports scalable, real-time processing of billions of vectors with advanced indexing techniques for fast and accurate results.

Contents

1. Installation
2. Configuration
3. Implementation
4. Usage
5. Troubleshooting

---

Installation

Step 1: Download Milvus

• Install Milvus locally or use Docker.
• For Docker:

docker pull milvusdb/milvus:standalone-latest
docker run -d --name milvus -p 19530:19530 -p 9091:9091 milvusdb/milvus:standalone-latest

• You will see successful download of milvus as follows:

Step 2: Install Python Dependencies

• Install required libraries using pip:

  pip install pymilvus sentence-transformers langchain

• You can see the installation of dependencies once you run the command:

Step 3: Set Environment Variables

• Create a .env file with the following content:

  MILVUS_URI=/app/milvus/milvus_vector.db
  GROQ_API_KEY=<your_api_key>

Step 4: Verify Connection

• Use the connections.connect function in your script to ensure Milvus is reachable:

  from pymilvus import connections
  connections.connect("default", uri="/app/milvus/milvus_vector.db")

Step 5: Test Installation

• Run a simple query to ensure Milvus is functioning correctly:

  from pymilvus import utility
  print(utility.list_collections())

---

Configuration

Step 1: Define the Vector Schema

• Create a schema for the Milvus collection:

  from pymilvus import CollectionSchema, FieldSchema, DataType
  
  fields = [
      FieldSchema(name="hash_id", dtype=DataType.VARCHAR, max_length=32, is_primary=True),
      FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=384),
      FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=5000),
      FieldSchema(name="title", dtype=DataType.VARCHAR, max_length=200),
      FieldSchema(name="source", dtype=DataType.VARCHAR, max_length=200),
      FieldSchema(name="dynamically_generated", dtype=DataType.BOOL)
  ]
  schema = CollectionSchema(fields, description="Vector store schema")

Step 2: Initialize the Collection

• Create a collection in Milvus using the schema:

  from pymilvus import Collection
  collection = Collection(name="MyCollection", schema=schema)

Step 3: Set Indexing Parameters

• Create an index for the embedding field to improve search performance:

  collection.create_index(
      field_name="embedding",
      index_params={
     "index_type": "HNSW",
     "metric_type": "IP",
     "params": {"M": 16, "efConstruction": 200}
     })

Step 4: Search Parameter Configuration

• Define the parameters for querying the collection:

  search_params = {
      "metric_type": "IP",
      "params": {
         "ef": 200
      }
  }

Step 5: Verify Configuration

• Use the utility.list_collections() function to ensure the collection is registered:

  from pymilvus import utility
  print(utility.list_collections())

  

---

Implementation

Step 1: Load Documents

• Fetch and clean documents from the web using RecursiveUrlLoader:

  

Step 2: Clean and Split Documents

• Clean the documents to remove unnecessary HTML elements:

  • Fetch the content from the main section of the webpage

  • Clean the data and remove extra spaces and empty lines from the main content

• Split documents into chunks for vectorization:

  

Step 3: Generate Embeddings

• Use SentenceTransformers to encode document text into embeddings:

  • Generate the embedding model based on the SentenceTransformers model

  • Embed each chunk of cleaned data and insert into the vector store

Step 4: Insert Data into Milvus

• Add document chunks and embeddings to Milvus using the create_vector_store function:

  

Step 5: Test Querying

• Search for relevant documents using a query embedding:

  query_embedding = model.encode("example query")
  results = collection.search(
      data=[query_embedding],
      anns_field="embedding",
      limit=5,
      output_fields=["text", "title", "source"]
  )

---

Usage

Step 1: Initialize Milvus Connection

• Establish a connection to the Milvus database:

Step 2: Retrieve Relevant Documents

• Use the ScoreThresholdRetriever to retrieve documents based on query embeddings:

from retriever import ScoreThresholdRetriever

Step 3: Generate Responses

• Combine retrieved documents and generate an answers:

Step 4: Synchronize updates

• Update the Milvus collection by comparing hashes of new and existing documents:

def update_collection(docs, existing_hashes, collection):
   documents_to_insert = [
      doc for doc in docs if hash_text(doc.page_content) not in existing_hashes
   ]
   collection.insert(documents_to_insert)
   collection.load()

Step 5: Monitor Performance

• View logs to verify real-time operation during usage:

import logging
logging.basicConfig(level=logging.INFO)
logging.info("Milvus connection established and ready for querying.")

---

Troubleshooting

Connection Issues

• Problems connecting to the Milvus server, resulting in timeout errors or inability to access the vector store
  • Verify Milvus Server Status: Run docker ps to check if the Milvus container is active.
  • Confirm Network Access: Use ping or curl to check access to the Milvus server.
  • Firewall Configuration: Ensure no firewall rules block the server port.

Security considerations

• HTTPS Configuration: Serve the Streamlit app over HTTPS, especially in production, using reverse proxies like Apache or Nginx.

Embedding Retrieval Failures

• We may encounter errors when attempting to retrieve embeddings or receive empty results
  • Confirm that embeddings have been successfully inserted into the vector store by checking the database status
  • Ensure that the retrieval query is correctly structured, and the vector dimensions match those stored in the database
  • Adjust the search parameters such as score thresholds to allow for more flexible retrieval options

Version Compatibility Problems

• Issues may arise if there is a mismatch between the PyMilvus library version and the Milvus server version
  • Ensure your PyMilvus version matches the Milvus server version using Milvus Compatibility Matrix
  • Update the PyMilvus library using pip install --upgrade pymilvus to ensure you have the latest features and fixes

Performance Issues

• Slow response times during embedding storage or retrieval may occur, affecting the chatbot's responsiveness
  • Optimize data ingestion by using batching for bulk operations to improve efficiency
  • Monitor server resources (CPU, memory, and disk usage) to identify bottlenecks
  • Tune Milvus configuration settings like insert_timeout and search_timeout to better suit your workload

Debugging Logs

• If problems persist, users can refer to the Milvus server logs for detailed error messages and diagnostics
  • Access log files located in the Milvus installation directory (usually located at /var/log/milvus/)
  • Review the logs to identify specific errors or warnings that can provide insight into the issues being faced