Building RAG with Milvus Vector Database

Building RAG with Milvus Vector Database

When Millions of Chunks No Longer Fit in RAM

Relational databases with cosine distance fail at volumes >100K vectors. We encountered this in a fintech project: 2.5M documentation chunks, 6 languages, requirement P99 latency <500 ms. We chose Milvus — an open-source vector DB with HNSW indexes and GPU acceleration. We'll share how we tuned hybrid search (dense + sparse) in production.

A typical problem is "phantom" relevant results: the model returns semantically similar but contextually irrelevant items. We solved this with hybrid search using RRF reranking. Our experience shows that proper index configuration reduces latency by 40%.

Problems We Solve

Scale: 2.5M chunks is not the limit. Milvus scales horizontally easily: we deployed clusters up to 10 nodes supporting billions of vectors. Choosing the right index is key: for our client we used HNSW (M=32, efConstruction=400) for dense and SPARSE_INVERTED_INDEX for sparse.

Hybrid search: dense vs sparse. Dense vectors capture semantics well but miss exact keyword matches. We combined both approaches with RRF reranking. Relevance increased by 25%.

Multi-tenancy without headaches. Partitioning isolates client data without creating separate collections. We implemented a dynamic partition scheme — easy and secure.

Why Milvus Is the Best Choice for Enterprise RAG?

Milvus outperforms other vector DBs in speed and cost under high load. Compare indexes:

We use HNSW as a universal choice for production, and DISKANN for archival data on SSD.

Why Milvus Is Better Than Pinecone for High Loads?

Milvus is cheaper at volumes >1M vectors since it doesn't charge per query. Compare:

For a typical 10M vector cluster, Milvus is 3-5 times cheaper than Pinecone with the same performance.

How to Configure HNSW Indexes for Minimal Latency?

Key parameters: M (16-64) and efConstruction (200-500). Higher efConstruction means better accuracy but slower building. In production we use M=32, efConstruction=400, and for search ef=100-200. This gives P99 latency <400 ms on 2.5M chunks.

How We Configured Hybrid Search for High Relevance?

Here's the code we applied for our client (fintech, 2.5M chunks):

from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType, utility

# Connect to Milvus
connections.connect(
    alias="default",
    host="localhost",
    port="19530"
)

# Or via URI (Milvus Lite for local development)
from pymilvus import MilvusClient
client = MilvusClient("./milvus_local.db")  # SQLite-like file

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=4096),
    FieldSchema(name="source", dtype=DataType.VARCHAR, max_length=512),
    FieldSchema(name="doc_type", dtype=DataType.VARCHAR, max_length=64),
    FieldSchema(name="page", dtype=DataType.INT32),
    FieldSchema(
        name="dense_vector",
        dtype=DataType.FLOAT_VECTOR,
        dim=1536  # text-embedding-3-small
    ),
    FieldSchema(
        name="sparse_vector",
        dtype=DataType.SPARSE_FLOAT_VECTOR  # BM25
    ),
]
schema = CollectionSchema(fields=fields, description="Corporate Knowledge Base")
collection = Collection(name="knowledge_base", schema=schema)
# Indexes for vector fields
collection.create_index(
    field_name="dense_vector",
    index_params={"metric_type": "COSINE", "index_type": "HNSW", "params": {"M": 16, "efConstruction": 200}}
)
collection.create_index(
    field_name="sparse_vector",
    index_params={"metric_type": "IP", "index_type": "SPARSE_INVERTED_INDEX"}
)
collection.load()

from pymilvus import AnnSearchRequest, RRFRanker

def milvus_hybrid_search(query: str, top_k: int = 5) -> list:
    # Dense vector
    dense_vec = dense_embedder.embed_query(query)
    # Sparse vector (via built-in BM25Encoder)
    sparse_vec = sparse_encoder.encode_queries([query])

    # Two requests for RRF
    dense_req = AnnSearchRequest(
        data=[dense_vec],
        anns_field="dense_vector",
        param={"metric_type": "COSINE", "params": {"ef": 100}},
        limit=30,
    )
    sparse_req = AnnSearchRequest(
        data=sparse_vec,
        anns_field="sparse_vector",
        param={"metric_type": "IP"},
        limit=30,
    )

    # RRF fusion
    results = collection.hybrid_search(
        reqs=[dense_req, sparse_req],
        rerank=RRFRanker(k=60),
        limit=top_k,
        output_fields=["text", "source", "doc_type"],
    )
    return results

Result: 850 QPS with P99 latency <400ms on a 3-node cluster (8 vCPU, 32GB RAM each).

What's Included in the Work

• Data audit and collection schema design
• Milvus cluster deployment (Kubernetes / bare-metal) with GPU acceleration
• Index and parameter tuning (efConstruction, M, ef) for load
• Ingestion pipeline (PySpark / Kafka / direct)
• RAG pipeline with LangChain or LlamaIndex
• Documentation and team training
• 2 weeks of post-launch support

Process

1. Analysis: gather requirements, assess volume and query frequency.
2. Design: select schema, indexes, cluster topology.
3. Implementation: deploy Milvus, write pipelines, integrate with LLM.
4. Testing: load testing, parameter tweaks.
5. Deployment: CI/CD, monitoring (Prometheus + Grafana).

Timeline

• Milvus cluster setup + schema: 3–5 days
• Ingestion pipeline with hybrid indexing: 5–10 days
• RAG pipeline and evaluation: 1–2 weeks
• Total: 3–5 weeks

Company Metrics

Over 7 years working with vector databases, 30+ Milvus production deployments. We guarantee 99.9% uptime on the cluster and expert support. We'll evaluate your project in 2 days — contact us. Get a consultation on RAG architecture: we'll estimate cost and timeline for your data volume.