Automotive Technical Intelligence RAG System

Introduction

Large Language Models (LLMs) often hallucinate when queried about highly specific, version-dependent data, such as automotive technical specifications that change across model years. This project introduces a robust Retrieval-Augmented Generation (RAG) architecture designed to ground LLM responses exclusively in verified manufacturer documentation. By deploying a serverless hybrid search pipeline, the system effectively bridges the gap between unstructured web documentation and deterministic technical intelligence, ensuring high-fidelity answers while maintaining strict memory and compute constraints in a cloud environment.

Architecture & Pipeline

The system operates across two primary pipelines: Data Ingestion and Query Execution.

• Ingestion & Processing: Dynamic automotive web pages and PDFs are scraped using Playwright and BeautifulSoup, parsed into clean Markdown, and chunked contextually. Embeddings are generated and stored in a Supabase PostgreSQL cluster utilizing the pgvector extension.
• Hybrid Retrieval Strategy: The execution pipeline employs a dual-retrieval mechanism combining Dense Vector Search (pgvector) for semantic alignment and Sparse Lexical Search (BM25 via Postgres FTS) for exact keyword matches, unified via Reciprocal Rank Fusion.
• Cross-Encoder Reranking: Initial retrieved candidates are evaluated by an MS-MARCO Cross-Encoder model to score deep logical relevance against the user query before being passed to the Groq LLM API for final synthesis.
• Memory Optimization: To prevent Out-Of-Memory (OOM) failures in constrained cloud environments, standard PyTorch dependencies were replaced with the fastembed ONNX runtime, reducing the active RAM footprint from 550MB to under 100MB without accuracy degradation.

Results & Benchmarks

A comprehensive Continuous Integration (CI) pipeline was established via GitHub Actions to automate system evaluation. A ground-truth benchmark suite of diverse 100 queries was constructed to validate retrieval precision across varying levels of query complexity.

Findings

• Hybrid Superiority: Relying solely on dense embeddings failed when querying exact alphanumeric codes (e.g., "M256 engine"). Integrating BM25 lexical search successfully bridged this gap, demonstrating that hybrid retrieval is essential for technical documentation.
• ONNX Runtime Efficiency: Transitioning ML models from PyTorch to the ONNX runtime proved that production-grade semantic search and reranking can be efficiently executed within strict 512MB memory limits without utilizing external vector APIs.

Engineering Challenges & Architectural Solutions

Deploying a production-grade RAG pipeline within strict resource constraints (free-tier serverless and container runtimes) required resolving several critical engineering bottlenecks:

• The 512MB RAM Wall (PyTorch to ONNX): Standard NLP retrieval pipelines using torch and transformers require over 500MB of baseline memory, causing container crashes (OOMKilled) on constrained cloud hosts. PyTorch dependencies were replaced with the ONNX Runtime via fastembed, slashing baseline application memory from ~550MB to under 90MB without sacrificing retrieval precision.
• Schema Dimension Drift in PostgreSQL: Switching to lightweight models altered embedding dimensions from 768 to 384 dimensions. This required refactoring database initialization scripts in db.py to enforce VECTOR(384), dropping outdated vector tables with cascading constraints in Supabase, and rebuilding the hybrid search indexes (vector distance + full-text search tsvector) to guarantee mathematical alignment.
• Diagnosing "Ghost" CORS Failures & Cold Starts: The Next.js frontend frequently reported net::ERR_FAILED, mimicking a misconfigured CORS policy. Root-cause analysis revealed the errors were downstream symptoms of connection drops caused by serverless cold boots exceeding client timeout thresholds. This was resolved by implementing an external cron keep-alive monitor pinging the API every 10 minutes to maintain persistent active memory.
• Post-Inference Memory Spikes & Micro-Batching: The cross-encoder reranker initially loaded all top-20 document candidates into memory simultaneously for matrix multiplication, causing a post-inference memory spike that triggered silent kernel reboots. Three-stage memory throttling was implemented: reducing database retrieval from top_k=20 to top_k=10, throttling Cross-Encoder inference into micro-batches (batch_size=2), and injecting manual garbage collection (gc.collect()) directly after text generation to immediately purge ONNX C++ execution buffers from system RAM.

Limitations

• Data Mutability: The ingestion pipeline currently requires manual triggering to update the database when new automotive models are released, lacking a webhook-driven automated synchronization system.
• Generative Rate Limits: While the retrieval pipeline can handle high concurrency, the final answer generation is bottlenecked by the strict rate limits of the free-tier Groq API endpoint.