MCP + RAG Performance Stack: Tools, Benchmarks, and Decision Guide

TL;DR

• This is the technical follow-up to How to Make App Data Public and Production-Ready with MCP and RAG. Part 1 covered safety and architecture. This one covers what to pick and why.
• Do not choose tools from leaderboard screenshots alone. Use MTEB retrieval tasks for embedding selection, BEIR-style zero-shot retrieval for architecture comparisons, and your own eval set for final decisions.
• For most public-docs MCP + RAG stacks in 2026, my default is: hybrid retrieval (BM25 + dense), BGE-M3 or a strong API embedder, pgvector or Qdrant depending on ops model, FlashRank or BGE-Reranker-v2-m3, retrieval-set cache + prompt caching, and RAGAS + trace tooling for regression tests.
• Benchmarks disagree by workload. Timescale’s 50M-vector pgvector vs Qdrant study shows pgvectorscale winning on throughput while Qdrant wins on tail latency. Treat vendor claims as hypotheses until you rerun on your corpus size, filters, and hardware.
• The highest-leverage optimizations are usually boring: smaller top-k, hash-based re-embedding, retrieval-set caching, cascade reranking, and measuring recall@k before tuning the LLM.

What You Will Learn Here

• Which benchmarks actually help you choose retrieval components
• A decision flow for embeddings, search backends, rerankers, caches, and eval tools
• Open-source libraries worth knowing in 2026
• Comparison tables with benchmark context and production caveats
• Three reference stacks: MVP, balanced production, and high-scale
• How MCP serving choices affect latency and cache design
• A minimal benchmark harness you can run on your public corpus

How This Fits Part 1

Part 1 answered: What should be public? How do MCP and RAG fit together? Where are the safety boundaries?

This article answers: Given that architecture, which components make it fast, cheap, and measurable?

If you have not read part 1, skim its caching and checklist sections first. Everything here assumes a public allowlisted corpus and a retrieval path shared by MCP tools and RAG answers.

The Decision Flow

Start with constraints, not brands.

Constraints
  corpus size, QPS, p95 budget, budget $/1k queries,
  self-host vs managed, metadata filters, languages
        |
        v
Retrieval quality target
  recall@20 on your eval set
        |
        v
Pick first-stage retrieval
  BM25 only? dense only? hybrid? BGE-M3 multi-function?
        |
        v
Pick search backend
  pgvector, Qdrant, OpenSearch, in-memory FAISS for MVP
        |
        v
Pick reranker tier
  none -> FlashRank CPU -> BGE-v2-m3 GPU -> managed API
        |
        v
Pick cache layers
  CDN, embedding cache, retrieval-set cache, semantic cache, prompt cache
        |
        v
Pick eval + tracing
  RAGAS, custom recall@k, LangWatch/Langfuse/Phoenix
        |
        v
Load test + cost model
  p50/p95 latency, $/query, cache hit rate, recall drift

The rest of this article walks that tree with tools, numbers, and tradeoffs.

Benchmarks You Should Trust — and How to Misread Them

Benchmarks are useful for shortlisting, not for final sign-off.

MTEB and BEIR

MTEB (Massive Text Embedding Benchmark) is the standard embedding comparison surface. BEIR’s retrieval tasks live inside MTEB as zero-shot domain-transfer tests. The Hugging Face MTEB leaderboard is the practical place to compare models, but filter to Retrieval tasks when choosing embedders. Aggregate MTEB scores hide retrieval weakness.

BEIR originally framed retrieval as comparable across lexical, sparse, dense, late-interaction, and reranking architectures under one task format. That is still the right mental model even if you never run the full 18-dataset suite yourself.

Practical rule:

BRIGHT, RAGBench, CRAG, RGB

Use these when MTEB/BEIR are too optimistic:

• BRIGHT — reasoning-heavy retrieval; useful if users ask multi-step product questions.
• RAGBench / RAGChecker — component-level pipeline diagnosis.
• CRAG / RGB — end-to-end factual QA and refusal behavior; summarized well in RAG Apps in Practice.

My editorial judgment: public product docs usually fail on entity match, version drift, and missing evidence before they fail on exotic reasoning. Run BEIR-like retrieval tests first, then add 20 adversarial cases from part 1’s checklist.

retrieval-pareto and HAKARI-Bench

Two 2026 resources are especially useful for engineers who want quality/latency/storage tradeoffs, not just model names:

• retrieval-pareto — open benchmark harness comparing dense HNSW, OPQ-IVF-PQ, RaBitQ, ScaNN, BM25, late-interaction, and RRF hybrids across BEIR/BRIGHT-style tasks with p50/p99 latency and storage.
• HAKARI-Bench — lightweight benchmark designed to compare retrieval architectures and efficiency settings under unified conditions, including rerankers over a fixed candidate set.

Important caveat from retrieval-pareto itself: on some BEIR-scale corpora, exact flat dense search can beat HNSW because matrix math beats graph traversal overhead at moderate scale. Do not assume HNSW is automatically faster.

Leaderboard score                Your production question
-------------------              ---------------------------
MTEB retrieval nDCG@10     ->    recall@20 on *your* docs
BEIR average               ->    zero-shot on *your* domain
Vendor p50 latency chart   ->    p95 with *your* filters + reranker
Blog "73% cost reduction"  ->    cache hit rate on *your* traffic

Layer 1: Embedding Models

What the benchmarks suggest

On public English product docs, strong 2025–2026 options include:

The BGE-M3 paper (Chen et al., 2024) is the primary source for hybrid retrieval in one embedder: dense, lexical sparse, and ColBERT-style multi-vector, combined with weighted hybrid ranking:

s_rank = w1 * s_dense + w2 * s_lex + w3 * s_mul

For public docs with SKUs, API paths, error codes, and policy numbers, hybrid retrieval is usually worth it. Dense-only misses exact tokens; sparse-only misses paraphrases.

Decision guide

Corpus < 100k chunks, English, budget-sensitive
  -> e5-base or embedding-3-small + BM25 hybrid

Corpus multilingual or long docs (8k tokens)
  -> BGE-M3 self-host or strong multilingual API embedder

Need best quality, low eng time
  -> API embedder + BM25 side index

Need lowest $ at scale
  -> small open embedder + hash skip on unchanged chunks + retrieval cache

Open-source libraries

Minimal BGE-M3 hybrid example:

from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)

def embed_for_index(texts: list[str]):
    return model.encode(
        texts,
        return_dense=True,
        return_sparse=True,
        return_colbert_vecs=False,  # enable only if you need multi-vector
    )

Layer 2: Search Backends and Index Types

This is where benchmarks diverge most. Your ops model matters as much as raw speed.

pgvector vs Qdrant vs others

The most cited open comparison in 2025–2026 is Timescale/Tiger Data’s 50M-vector, 768-dim benchmark using a fork of ANN-Benchmarks:

Source: Timescale pgvector vs Qdrant comparison.

Interpretation for MCP + RAG over public docs:

• Choose pgvector (+ pgvectorscale at large scale) when you already run Postgres, want SQL metadata joins, ACID, one database for chunks + eval tables + app data, and high throughput matters.
• Choose Qdrant when you want a dedicated vector service, lower tail latency, simpler horizontal scaling for vector-only workloads, or rich filtering/payload patterns without fighting SQL planner quirks.
• Choose OpenSearch/Elasticsearch when lexical search is first-class and hybrid is non-negotiable in one cluster.
• Choose FAISS in-process only for MVPs, offline jobs, or embedded search in a single worker — not as your multi-tenant serving tier.

                    OPS PREFERENCE
                          |
         +----------------+----------------+
         |                                 |
    Already on Postgres              Vector-first team
         |                                 |
         v                                 v
   pgvector HNSW                   Qdrant / Weaviate
   + pgvectorscale @ 10M+          + FastEmbed
         |                                 |
         +----------------+----------------+
                          |
                   Need lexical in one box?
                          |
                    OpenSearch hybrid

Index type cheat sheet

From retrieval-pareto and ANN-Benchmarks culture:

My practical recommendation for public docs under ~2M chunks: BM25 + pgvector HNSW or Qdrant hybrid payload index, then rerank top 20–50.

Layer 3: Rerankers

First-stage retrieval should optimize recall@50–100. Rerankers optimize precision@3–5.

Benchmark-backed options

Cross-encoders typically add ~5–15 nDCG points over bi-encoder-only retrieval on BEIR-style tasks. That matches the two-stage pattern in the original BEIR reranking setup (rerank top 100 BM25 hits).

FlashRank’s README is explicit: no Torch required, ~4MB default model, CPU-first. That makes it the best open-source choice when p95 budgets are tight and you can accept a quality step-down.

BGE-Reranker-v2-m3 is the best general-purpose self-hosted cross-encoder for teams with a small GPU or batch rerank worker.

Cascade reranking — the performance pattern most teams skip

Stage 0: hybrid retrieve top 100   (~20-80 ms)
Stage 1: FlashRank -> top 20       (~15-30 ms CPU)
Stage 2: BGE-v2-m3 -> top 5        (~40-90 ms GPU)
Stage 3: pack 3-4 chunks into LLM  (~5 ms)

This often beats “one huge reranker on 100 pairs” on cost and latency while preserving most quality gains. Validate with recall@5 and answer faithfulness, not reranker score alone.

from flashrank import Ranker, RerankRequest

ranker = Ranker(model_name="ms-marco-MiniLM-L-12-v2", cache_dir="/tmp/flashrank")

def rerank_stage1(query: str, passages: list[dict], top_n: int = 20):
    req = RerankRequest(query=query, passages=passages)
    return ranker.rerank(req)[:top_n]

Layer 4: Caching — Where the Money Actually Goes

Part 1 introduced four cache layers. Here is the tool mapping.

Cache layer matrix

GPTCache (MIT, Zilliz) is still the most flexible open-source semantic cache library: pluggable embedder, vector store, eviction, LangChain/LlamaIndex integrations. Primary reference: GPTCache ACL paper.

Important implementation detail — repeated from part 1 because benchmarks ignore it: cache retrieval sets or grounded answer packages, not only free-form LLM text, if you care about citation stability.

Semantic cache comparison

Suggested stack order for public docs traffic:

1. CDN for static docs + MCP read-only resources
2. Retrieval-set cache (15 min TTL, corpus-version keyed)
3. Provider prompt caching on stable system prefix
4. Semantic cache for top 100 FAQ queries
5. LLM generation

Corpus version in the cache key is non-optional:

CACHE_KEY = f"{corpus_version}:{hash(query)}"

Layer 5: Orchestration Libraries

You do not need a heavy framework, but you do need consistent retrieval traces.

My practical judgment: for MCP tool handlers that must stay fast, keep orchestration thin inside the tool:

MCP tool -> retrieval service -> cache -> vector store
          (no LLM in the tool path unless explicitly requested)

Put LLM synthesis in a separate RAG endpoint or a different MCP tool (answer_public_question) so agents can choose search-only vs full answer.

Layer 6: Eval and Observability Tools

Performance work without metrics becomes vibe tuning.

Evaluation libraries

RAGAS is the default open-source starting point because it separates retrieval usefulness from generation quality without requiring gold answers for every metric. Primary source: Es et al., EACL 2024 demo.

Minimal retrieval benchmark harness:

from dataclasses import dataclass

@dataclass
class RetrievalExample:
    question: str
    gold_chunk_ids: set[str]

def recall_at_k(ex: RetrievalExample, retrieved_ids: list[str], k: int = 20) -> float:
    if not ex.gold_chunk_ids:
        return 1.0
    return len(ex.gold_chunk_ids & set(retrieved_ids[:k])) / len(ex.gold_chunk_ids)

def benchmark_retriever(retriever, examples: list[RetrievalExample], k: int = 20) -> dict:
    scores = [recall_at_k(ex, retriever.search(ex.question, k=k), k) for ex in examples]
    return {"recall_at_k": sum(scores) / len(scores), "k": k, "n": len(examples)}

Run this whenever you change embedder, index params, hybrid weights, or reranker.

Metrics that matter for MCP + RAG

MCP-Specific Performance Choices

MCP serving details affect the retrieval stack.

Keep search tools deterministic and cacheable

From the MCP 2026-07-28 release candidate (blog post):

• protocol-level sessions are going away on Streamable HTTP
• list/read responses gain ttlMs and cacheScope cache hints

Design implications:

search_public_docs(query, product, limit)
  -> cacheable, deterministic, no LLM
  -> set short ttlMs on tool list/metadata
  -> key cache by (corpus_version, normalized_query, filters)

answer_public_question(query)
  -> higher cost, lower cache hit rate
  -> separate tool with stricter rate limits

Stateless handlers, explicit IDs

Pass corpus_version, product, and locale as ordinary tool args. Do not rely on server session memory for tenant or corpus selection.

Transport

For remote public MCP:

• prefer Streamable HTTP behind a reverse proxy
• set aggressive timeouts on tool calls
• co-locate MCP server and retrieval service in the same region as the vector store

Three Reference Stacks

These are templates, not commandments. Run the benchmark harness on your corpus before committing.

Stack A — MVP (fast to ship)

Static docs + llms.txt
Postgres + pgvector HNSW
BM25 via Postgres FTS or Meilisearch sidecar
e5-base-v2 or embedding-3-small
No reranker (or FlashRank on top 30 only)
In-memory retrieval-set cache
RAGAS offline eval on 50 examples

Good for: startup public docs, <200k chunks, low QPS, one engineer.

Weak under: high QPS, multilingual, heavy SKU/code lookup miss rate.

Stack B — Balanced production (my default)

Public corpus in Postgres metadata + pgvector
BM25 + BGE-M3 dense hybrid
FlashRank cascade -> BGE-Reranker-v2-m3
Retrieval-set cache in Redis (corpus-version keyed)
Semantic cache via GPTCache for top queries
MCP tools: search_public_docs + answer_public_question
RAGAS + custom recall@k in CI
Langfuse or LangWatch traces in prod
Provider prompt caching on stable system prefix

Good for: public product docs, integrations catalogs, developer portals.

Weak under: billion-vector scale without pgvectorscale/Qdrant sharding plan.

Stack C — High scale / strict p95

Qdrant hybrid + payload filters
BGE-M3 or top API embedder
Two-stage rerank: FlashRank -> mxbai-rerank-large-v2 or Cohere API
Redis retrieval-set cache + CDN for MCP resources
Dedicated rerank GPU workers
retrieval-pareto-style load tests before launch
Component evals + canary corpus versions

Good for: high QPS public search, multi-product filters, global latency SLOs.

Weak under: small teams without ops capacity for another data store.

Comparison Tables at a Glance

Embeddings (retrieval-first)

*pair with BM25 or OpenSearch for hybrid.

Vector stores (public docs scale)

Rerankers

Load Test Checklist

Before calling the performance stack production-ready:

• recall@20 >= your target on 50+ labeled questions
• reranker improves precision@5 measurably vs no reranker
• p95 MCP search_public_docs within SLO at expected QPS
• cache hit rate measured after 1 week of shadow traffic
• corpus reindex does not stall queries (blue/green or versioned index)
• embed cost per reindex run documented with hash-skip enabled
• regression CI fails on retrieval metric drop > agreed threshold

What Not to Overclaim

• Leaderboard nDCG is not your FAQ accuracy.
• Semantic caching savings depend on query repetition; developer docs often have long tail queries with low cache hit rates.
• A reranker cannot fix recall failures — if gold chunks are outside top 50, fix retrieval first.
• MCP cache hints (ttlMs) help clients, but you still need server-side retrieval-set caching for real cost wins.
• Managed rerank APIs can win benchmarks while losing p95 once network variance is included — measure end-to-end from MCP client if that is the user path.

Sources

• How to Make App Data Public and Production-Ready with MCP and RAG — architecture and safety baseline (part 1)
• RAG Apps in Practice: Structured Knowledge, Better Retrieval, and Real Evals — RAG chain design and CRAG/RGB context
• MTEB leaderboard — embedding and reranking comparisons
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021
• BGE M3-Embedding paper — Chen et al., 2024; hybrid dense/sparse/multi-vector retrieval
• BGE-M3 documentation — hybrid ranking formula and usage
• retrieval-pareto — open retrieval quality/latency/storage benchmarks
• HAKARI-Bench — lightweight unified retrieval architecture benchmark
• Timescale: pgvector vs Qdrant at 50M vectors — throughput vs tail latency comparison
• FlashRank — CPU reranker library
• BGE-Reranker-v2-m3 model card — self-hosted cross-encoder default
• GPTCache — open-source semantic cache library
• GPTCache ACL paper — semantic caching for LLM applications
• RAGAS — RAG evaluation framework
• RAGAS: Automated Evaluation of RAG — Es et al., EACL 2024
• The 2026-07-28 MCP Specification Release Candidate — stateless HTTP, cache hints
• Anthropic: Prompt caching — provider-side input token savings
• OpenAI: Prompt caching — provider-side input token savings