How to Make App Data Public and Production-Ready with MCP and RAG

TL;DR

• Making app data “public” is not the same as dumping your database into a vector store. It is a deliberate allowlist of docs, catalogs, changelogs, pricing pages, and bounded read APIs that agents and humans can trust.
• The production pattern that works in 2026 is three layers: discoverable static content (llms.txt, markdown, sitemaps), an MCP server for structured resources and bounded tools, and a RAG service for semantic search over the same public corpus.
• Safety comes from separation: public read path, authenticated write path, tool-level scopes, and retrieval filters that never touch private rows. OAuth 2.1 is for protected MCP endpoints, not for everything.
• Cost control is a stack, not one trick: cache embeddings and retrieval sets, use provider prompt caching for stable system prompts, semantic-cache frequent questions, route easy queries to smaller models, and cap context size.
• MCP is moving toward better native caching (ttlMs, cacheScope in the 2026-07-28 release candidate). Plan for HTTP Streamable transport, stateless servers, and evals before you call the system production-ready.
• If you already have agent-ready website infrastructure and RAG design basics, this article is the bridge: how to ship your app’s public knowledge as a safe, cheap, agent-facing product surface.

What You Will Learn Here

• What “public app data” should mean for PMs and engineers
• A reference architecture for MCP + RAG over a public corpus
• How to draw the safety boundary between public reads and private writes
• MCP server design choices: resources vs tools, auth, caching, server cards
• RAG patterns that stay safe, fast, and cheap at scale
• A concrete public-docs assistant example with code sketches
• A production checklist you can run before launch

Inferred Article Shape

This article assumes you want to expose product knowledge to external agents and users without opening your whole backend. The sections below move from PM framing → architecture → safety → MCP → RAG → example → checklist. If your goal is only internal employee RAG, skip the public allowlist sections and keep the MCP/RAG production guidance.

The PM Question: What Should Be Public?

Before MCP or RAG, answer a product question:

Which parts of our app data help users and agents make better decisions when shared openly?

Good public candidates:

• product docs, API references, SDK guides
• pricing pages, plan limits, feature matrices
• public changelogs, release notes, status pages
• open catalogs: integrations, templates, public datasets, marketplace listings
• support articles marked for public consumption
• bounded read APIs: “search public docs”, “get plan limits”, “list public integrations”

Bad public candidates:

• customer records, billing history, support tickets with PII
• internal runbooks, incident notes, employee directories
• raw database exports “because embeddings need text”
• admin tools, write APIs, or anything that mutates state without strong auth

Use a simple gate:

My editorial judgment: most teams fail here by indexing everything searchable instead of defining a public corpus with owners, TTLs, and review rules.

Three Layers, One Public Corpus

Think in three layers that share one allowlisted source of truth:

Layer 1: Discoverable Content
  llms.txt, markdown pages, sitemap, /openapi or API catalog
  -> passive reading for crawlers and agents

Layer 2: MCP Server
  resources (read structured public data)
  tools (bounded queries: search_docs, get_pricing, list_integrations)
  -> active capability discovery and tool calls

Layer 3: RAG Service
  hybrid retrieval + rerank + citations over the same public index
  -> semantic answers with evidence and refusal behavior

Why three layers instead of one?

• Static content is the cheapest surface. Agents can read it without your infra running a model.
• MCP gives host clients a standard way to discover and invoke capabilities. The MCP specification defines servers that expose resources, prompts, and tools over JSON-RPC.
• RAG helps when users ask natural-language questions that span many pages, need ranking, or require synthesized answers with citations.

The mistake is building only one layer. Docs alone are hard to query. RAG alone is hard to discover and audit. MCP alone does not replace search quality for fuzzy questions.

                    +------------------+
                    |  Public corpus   |
                    |  (allowlisted)   |
                    +--------+---------+
                             |
           +-----------------+-----------------+
           |                 |                 |
           v                 v                 v
    +-------------+   +-------------+   +-------------+
    | Static web  |   | MCP server  |   | RAG index   |
    | llms.txt    |   | resources   |   | chunks +    |
    | *.md pages  |   | tools       |   | metadata    |
    +------+------+   +------+------+   +------+------+
           |                 |                 |
           v                 v                 v
    Crawlers /          Agent hosts /       Your app chat /
    research agents     Claude, Cursor      support bot

For discoverability foundations, see Agent-Ready Infrastructure. This article focuses on the app data side once you accept that agents will read and call your product.

Reference Architecture

Here is a production-shaped default for a SaaS product exposing public knowledge:

Public sources                Ingestion                     Serving
-------------                 --------                      -------

/docs/*.md          ----->    Normalize + frontmatter  ---> Static CDN
/api/openapi.yaml            Extract stable IDs            llms.txt
/pricing.json                Chunk + hash                  markdown routes
/changelog/*.md              Embed changed chunks only     MCP HTTP server
/public/catalog.json         Metadata: product, version    RAG API
                             Freshness timestamps          Search UI

                             +-------- cache tier ---------+
                             | embedding cache             |
                             | retrieval-set cache         |
                             | semantic Q->docs cache      |
                             | provider prompt cache       |
                             +-----------------------------+

                             +-------- safety tier --------+
                             | public allowlist only       |
                             | auth on write tools         |
                             | scope checks per tool       |
                             | evals + refusal tests       |
                             +-----------------------------+

Design rules:

1. One allowlist, many surfaces. Do not maintain separate “website docs” and “RAG docs” that drift apart.
2. Compute behind a gateway. Browser, agent host, and MCP client all hit your API layer—not your database directly.
3. Separate read and write MCP tools. Public reads can be unauthenticated or lightly rate-limited. Writes require OAuth and tool scopes.
4. Measure the chain. Retrieval recall and citation quality matter as much as answer fluency. See RAG Apps in Practice.

Safety: Public by Design, Not by Accident

“Public” is a security property. Treat it like one.

1. Build a public allowlist, not a private denylist

Ingestion should read from explicit paths or tables:

/content/public/**
/openapi/public.yaml
/pricing/plans.public.json

Never point the indexer at production DB replicas unless rows are pre-filtered into a public export job.

2. Split MCP resources and tools by risk

The MCP authorization tutorial is explicit: authorization secures sensitive resources and operations. Authorization is optional in MCP, but if your server handles user data or admin actions, use OAuth 2.1 with HTTPS in production.

For protected endpoints, MCP servers should implement OAuth 2.0 Protected Resource Metadata at /.well-known/oauth-protected-resource, and clients discover the authorization server from that document. Enforce scopes at the tool level, not only at the server front door. See Building an MCP Server with Auth0 Login in Go and OpenFGA for Granular Permissions for deeper auth modeling.

3. Filter before retrieval, not after generation

If a chunk should not be public, it should not be embedded.

Bad:  retrieve top-k -> prompt "ignore private data"
Good: metadata.is_public = true filter -> retrieve -> generate

For mixed workspaces, use object-level authorization in the retrieval path. OpenFGA or equivalent checks belong before chunks enter the model context.

4. Add refusal and provenance requirements

Public RAG should:

• cite stable source IDs and URLs
• refuse when evidence is missing
• prefer exact docs for pricing, limits, and compliance facts
• label synthesis clearly when combining multiple pages

Benchmarks like CRAG and RGB keep showing that naive RAG improves answers but still fails on noise robustness, stale facts, and “I don’t know” behavior. Build evals for those cases before launch.

5. Rate-limit and monitor exfiltration patterns

Public endpoints will be scraped. Plan for:

• per-IP and per-token rate limits
• anomaly alerts on broad enumeration queries
• logging of tool calls without logging secrets or JWT bodies
• cache keys that do not leak tenant identifiers into shared public cache entries

Building the Public Corpus

Production RAG starts with ingestion discipline.

Stable IDs and metadata

Every public document should carry:

id: docs/api/authentication
source_url: https://example.com/docs/api/authentication.md
product: platform
visibility: public
version: 2026-06-01
updated_at: 2026-06-28T10:00:00Z
content_hash: sha256:...

Stable IDs make citations, eval datasets, and cache invalidation possible.

Chunk with structure preserved

Do not flatten tables, code blocks, or policy hierarchies. Prefer:

• parent/child chunks for long docs
• section-aware splits
• metadata for heading path (h1 > h2 > h3)
• separate chunks for pricing tables and limits

This mirrors the guidance in RAG Apps in Practice: preserve the knowledge structure users actually ask about.

Re-embed only when content changes

Hash each chunk at ingest time. If content_hash is unchanged, skip embedding calls. At scale, this is one of the cheapest wins in the pipeline.

Freshness tiers

Not all public data ages the same way:

Expose freshness in MCP resource metadata and RAG answers when the answer depends on time-sensitive facts.

MCP Server Design for Production

MCP is the contract between your app and agent hosts like Claude, Cursor, ChatGPT, and VS Code.

Resources vs tools

Use resources when the client should read structured public context:

• public://docs/getting-started
• public://pricing/plans
• public://changelog/2026-06-30

Use tools when the agent should invoke a bounded operation:

• search_public_docs(query, product, limit)
• get_integration(slug)
• list_public_templates(category)

Keep tool inputs narrow. Prefer structured outputs with source IDs over free-form prose.

Server cards and discovery

Publish a machine-readable MCP server card and link it from llms.txt or your docs footer. Few products do this today, which is exactly why early adopters gain distribution in agent hosts. The discoverability patterns are covered in Agent-Ready Infrastructure.

Transport and deployment

For remote public servers:

• prefer HTTP Streamable transport for production
• keep the MCP server stateless; pass explicit IDs in tool args
• place it behind a reverse proxy with TLS, health checks, and request timeouts
• separate the OAuth authorization server from the MCP resource server

The MCP specification’s 2026-07-28 release candidate, locked on May 21, 2026, moves the core protocol toward stateless HTTP and adds standardized cache hints (ttlMs, cacheScope) on list/read results. Final publication is targeted for July 28, 2026. If you are starting now, design for stateless handlers and explicit cache headers so you do not rebuild when clients adopt the new version.

Example: minimal public MCP tool schema

{
  "name": "search_public_docs",
  "description": "Search publicly available product documentation. Returns snippets and source URLs only.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": { "type": "string", "minLength": 2, "maxLength": 200 },
      "product": { "type": "string", "enum": ["platform", "sdk", "cli"] },
      "limit": { "type": "integer", "minimum": 1, "maximum": 5, "default": 3 }
    },
    "required": ["query"]
  }
}

The implementation should call your RAG retrieval service or search index—not loop an LLM inside the tool handler for every call.

RAG Layer: Safe, Optimized, Cached, Cheap

RAG over public data still needs production engineering.

Retrieval pipeline

A strong default for public product knowledge:

User question
     |
     v
Query rewrite (optional, small model)
     |
     v
Hybrid retrieval: BM25 + vector + metadata filters
     |
     v
Rerank top 20 -> keep top 3-5
     |
     v
Answer with citations + freshness note
     |
     v
Post-checks: grounding, policy, max tokens

Keep context small. Many teams retrieve ten chunks “to be safe” and pay for noise. For public docs, three to five good chunks usually beat ten mediocre ones.

Cache at four levels

Think beyond provider prompt caching:

L1  CDN / HTTP cache          static docs, llms.txt, MCP resource bodies
L2  Embedding cache           query/text -> vector
L3  Retrieval-set cache       query embedding -> chunk IDs + scores
L4  Semantic answer cache     similar question -> prior grounded answer

Provider prompt caching reduces cost for repeated long system prompts and stable retrieved context on supported APIs. Semantic caching reduces full LLM calls when questions repeat with different wording. GPTCache is a reference open-source pattern for semantic LLM caching; in production, many teams use Redis, pgvector, or their existing vector store for similarity lookup.

Practical note from field implementations: cache retrieval results or grounded answer packages (answer + source IDs), not only raw model text, if you use non-zero temperature and want consistent citations.

Set similarity thresholds deliberately. Public docs assistants often land around 0.90–0.94 cosine similarity for semantic cache hits, but validate on your own query logs. Lower thresholds save money and increase stale or wrong-answer risk.

Cost controls that actually ship

My practical judgment: teams that only optimize model choice without caching the retrieval path usually leave half the savings on the table.

Evals before “production-ready”

Minimum public RAG eval set:

• 50–100 real user questions
• 20 negative cases where the answer should refuse
• 20 freshness-sensitive cases (pricing, limits, deprecations)
• component metrics: retrieval recall@k, citation accuracy, groundedness, latency, cost per answer

If you expose the same corpus through MCP tools, add tool-level tests:

• tool schema validation
• scope enforcement for protected tools
• deterministic search tools return the same IDs for the same query
• rate-limit and timeout behavior

Example: Public Product Docs Assistant

Here is a compact end-to-end shape using Astro-style static content plus a server MCP/RAG path.

1. Publish the public corpus

content/public/
  docs/
    getting-started.md
    authentication.md
  pricing/plans.json
  changelog/2026-06-30.md
public/
  llms.txt
  openapi/public-api.yaml

Your static site serves markdown and llms.txt. The ingestion job mirrors the same files into the vector index.

2. Ingest with hashes

# ingestion/public_index.py
import hashlib
from dataclasses import dataclass

@dataclass
class PublicChunk:
    id: str
    text: str
    source_url: str
    metadata: dict
    content_hash: str

def hash_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def upsert_chunk(store, chunk: PublicChunk) -> None:
    existing = store.get_hash(chunk.id)
    if existing == chunk.content_hash:
        return  # skip re-embed
    vector = embed(chunk.text)
    store.save(chunk, vector)

3. Expose MCP search that reuses retrieval

# mcp/public_server.py
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("Public Docs")

@mcp.tool()
def search_public_docs(query: str, product: str = "platform", limit: int = 3):
    results = retrieval.search(
        query=query,
        filters={"visibility": "public", "product": product},
        limit=min(limit, 5),
    )
    return {
        "results": [
            {
                "id": r.id,
                "title": r.title,
                "url": r.source_url,
                "snippet": r.snippet,
                "updated_at": r.updated_at,
            }
            for r in results
        ]
    }

4. RAG answer path with retrieval-set cache

# rag/answer.py
def answer_public_question(question: str) -> dict:
    cache_key = semantic_key(question)
    cached = retrieval_cache.get(cache_key)
    if cached:
        chunks = cached
    else:
        chunks = retrieval.search(
            query=question,
            filters={"visibility": "public"},
            limit=4,
        )
        retrieval_cache.set(cache_key, chunks, ttl_seconds=900)

    if not chunks:
        return {"answer": "I could not find public documentation for that.", "sources": []}

    response = llm.generate(
        system=PUBLIC_DOCS_SYSTEM_PROMPT,
        context=format_chunks(chunks),
        question=question,
        max_output_tokens=300,
    )
    return {
        "answer": response.text,
        "sources": [{"id": c.id, "url": c.source_url} for c in chunks],
    }

5. Request flow

Agent host
   |
   |-- reads llms.txt + docs/*.md (free, cacheable)
   |
   |-- calls MCP tool search_public_docs
   |       -> retrieval-set cache
   |       -> hybrid search
   |
   '-- asks your RAG API for a synthesized answer
           -> retrieval-set cache
           -> rerank
           -> small/large model route
           -> citations

This keeps one public index behind both MCP and RAG while letting simple agents stay on static docs or tools only.

Production Checklist

Use this before calling the system production-ready.

PM / data governance

• Public corpus defined with named owners
• Legal/sales review for pricing, compliance, and partner content
• Freshness tiers documented
• “Not public” examples written down for support and engineering

Security

• Public allowlist enforced in ingestion and retrieval
• OAuth 2.1 on protected tools, HTTPS everywhere in production
• Tool-level scopes enforced
• No secrets, tokens, or private URLs in logs, prompts, or caches
• Rate limits and abuse monitoring enabled

MCP

• Resources and tools separated by risk
• Server card published and linked from docs
• Structured tool outputs with stable IDs
• Health checks and timeouts configured
• Cache headers or ttlMs policy documented for list/read endpoints

RAG

• Hybrid retrieval + metadata filters
• Chunk hashes to avoid unnecessary re-embedding
• Retrieval-set cache and semantic cache thresholds validated
• Provider prompt caching enabled where supported
• Eval set covers refusal, freshness, and citation accuracy
• Cost per answer and p95 latency monitored

Operational readiness

• Reindex runbook for doc publishes
• Cache invalidation tied to corpus version
• Dashboards for hit rate, error rate, spend, and retrieval recall
• Incident playbooks for stale pricing or policy answers

What Not to Overclaim

A balanced read matters for PMs and engineers:

• Public MCP plus RAG does not replace authenticated product APIs for customer-specific state.
• Semantic caching trades cost for staleness risk; tune thresholds and TTLs explicitly.
• MCP host support varies by client, transport, and auth flow. Test against the hosts your users actually use.
• The 2026-07-28 MCP changes are still moving through release candidate validation as of June 30, 2026. Build stateless servers now, but pin compatibility tests to the spec version you target.

Sources

• Model Context Protocol Specification (2025-11-25) — MCP base protocol, resources, tools, prompts
• MCP Authorization specification — OAuth 2.1, Protected Resource Metadata, scope handling
• Understanding Authorization in MCP — production-oriented OAuth guidance
• The 2026-07-28 MCP Specification Release Candidate — stateless HTTP, cache hints (ttlMs, cacheScope); locked May 21, 2026
• GPTCache: An Open-Source Semantic Cache for LLM Applications — semantic caching pattern for LLM apps
• CRAG and RGB benchmarks — summarized in RAG Apps in Practice
• Related internal guides:
  • Agent-Ready Infrastructure: llms.txt, MCP, robots.txt for AI
  • RAG Apps in Practice: Structured Knowledge, Better Retrieval, and Real Evals
  • Building an MCP Server with Auth0 Login in Go
  • OpenFGA for Granular Permissions in Modern Apps, MCP, and Agent Workflows