Semantic MCP & LLM Gateway

Introduction

Transitioning Large Language Models (LLMs) and autonomous agents from prototype to production introduces severe infrastructure bottlenecks: runaway token bills, redundant API calls, untrusted tool execution, and single-provider outages. These challenges are amplified when adopting the Model Context Protocol (MCP), connecting an LLM to 30+ enterprise tools bloats every single prompt with thousands of lines of JSON schema definitions, degrading both Time-to-First-Token (TTFT) and tool-calling precision on smaller models.

This project introduces a serverless Semantic MCP & LLM Gateway that acts as a high-throughput middleware layer between client applications and upstream AI providers. By combining sub-millisecond BM25/TF-IDF tool-schema pruning, character trigram semantic caching, complexity-based compute routing, and automated multi-provider failover, the gateway cuts MCP schema token overhead by 88.9% and resolves paraphrased queries in <0.15ms at zero compute cost.

Architecture & 5-Stage Pipeline

Every incoming request traverses a deterministic five-stage execution pipeline before reaching an upstream LLM:

• Stage 1 — Pre-Flight Security Guardrail (<0.03ms):
  • A zero-dependency ingress filter inspects both incoming user prompts and outgoing LLM-generated MCP tool arguments against compiled regex signatures for Path Traversal (LFI), destructive SQL injection, credential leaks, and adversarial system prompt overrides.
• Stage 2 — Vector Semantic Cache (<0.15ms):
  • Queries undergo synonym canonicalization, suffix stemming, and L2-normalized character 3-gram vectorization. If cosine similarity against recent traffic exceeds the 0.78 threshold, the gateway short-circuits the LLM entirely and returns the cached payload at $0 cost.
• Stage 3 — Semantic MCP Tool Pruner (<0.10ms):
  • Rather than injecting all 30 registered enterprise MCP tool schemas (2,208 tokens) into the context window, a custom BM25/TF-IDF inverted index ranks tool descriptions against the prompt and binds only the top-3 relevant tools (~230 tokens).
• Stage 4 — Complexity Scorer & Tier Router:
  • A syntactic and lexical heuristic evaluates prompt length, code/JSON structures, and algorithmic reasoning markers on a 0.0–1.0 scale, routing operational tasks to a fast 20B tier (openai/gpt-oss-20b) and deep architectural queries to a 120B reasoning tier (openai/gpt-oss-120b).
• Stage 5 — Multi-Provider Circuit Breaker & Sandboxed Execution:
  • If the primary provider experiences an outage (HTTP 503) or rate limit (HTTP 429), the circuit breaker trips and cascades across backup architectures (qwen/qwen3.8-27b and Google gemini-3.5-flash) with zero user-facing downtime.

Results & Benchmarks

The gateway was benchmarked across operational DevOps queries, adversarial red-team payloads, and simulated upstream provider outages.

Findings

• Small Model Tool-Calling Precision: Binding 30+ MCP tool schemas simultaneously caused 20B-class models to hesitate or misformat tool parameters. Pruning the context window to the top-3 relevant tools enabled openai/gpt-oss-20b to match 120B tool-calling accuracy at a fraction of the latency.
• Zero-Dependency Lexical Embeddings vs. Neural APIs: Using a external embedding API for semantic caching adds 80–150ms of network overhead per request. Computing L2-normalized character 3-gram and stemmed token vectors locally in Python achieved 0.93+ similarity detection on domain paraphrases in under 0.1ms.

Engineering Challenges & Architectural Solutions

Building a zero-cost, permanently available AI gateway on serverless infrastructure required solving several systems engineering challenges:

• Stop-Word Skew in BM25 Tool Ranking: Initial unstemmed token matching caused unrelated tools with verbose descriptions (such as huggingface_search_models) to rank in the Top-3 due to shared English stop words and morphological variants (e.g., "failed" vs. "failing"). Implementing stop-word filtering, snake_case sub-token expansion, and regex suffix stemming eliminated false-positive tool rankings.
• Cascading Upstream Quota & 503 Spikes: Free-tier cloud LLM endpoints frequently experience transient HTTP 503 high-demand spikes or regional 404/429 quota blocks. Rather than relying on a single backup model, the circuit breaker was engineered with a multi-stage fallback array that cascades across both model families and API providers before gracefully degrading.
• Hybrid Next.js & Python Serverless Routing: Deploying a unified TypeScript frontend and Python FastAPI backend on a single domain caused path-stripping collisions during production rewrites. Configuring environment-aware next.config.ts proxies alongside Vercel edge rewrite rules preserved full REST path integrity across local and cloud environments.

Limitations

• Ephemeral In-Memory Cache State: Because the default semantic cache resides in serverless worker memory to guarantee zero external database latency and cost, cache entries reset across cold starts unless paired with an external Redis store.
• Single-Turn Tool Execution Loop: The sandboxed MCP executor currently resolves single-hop operational tool calls deterministically rather than executing multi-turn recursive agent loops.