Introduction
Evaluating academic discourse requires both holistic critique and precise identification of rhetorical markers. Standard Large Language Models (LLMs) excel at summarizing tone but frequently hallucinate or fail at deterministic, token-level span extraction. This project introduces a hybrid Multi-Agent NLP Architecture designed to evaluate academic texts based on the Appraisal Framework.
Building upon recent computational linguistics methodologies, the system's discriminative branch is powered by a Data Augmented RoBERTa (DA-RoBERTa) architecture. By leveraging few-shot prompting and synthetic sentence generation to augment the training data, the custom sequence tagger achieved a highly competitive 0.7159 Weighted F1-score on complex academic discourse datasets. By running this DA-RoBERTa extraction natively alongside a LangGraph multi-agent LLM grading workflow, the system achieves research-grade rhetorical analysis entirely within strict free-tier cloud memory limits.
Architecture & Pipeline
The system operates across a decoupled frontend and a dual-branch machine learning backend.
- Frontend Parsing & Ingestion: A Next.js application dynamically parses academic files (PDF, Word, TXT) and routes the extracted raw text to the backend via a persistent TCP/HTTP tunnel.
- Generative LLM Workflow (LangGraph): The text enters a compiled state machine where parallel LLM Reviewer Agents independently analyze the text against specific system prompts. An Aggregator Agent synthesizes the isolated critiques, resolving contradictions, and outputs a formatted Executive Summary and Actionable Revisions markdown scorecard.
- Discriminative ML Extraction (spaCy & DA-RoBERTa): Simultaneously, the text bypasses the LangGraph state and is fed natively into a custom-trained Data Augmented RoBERTa (DA-RoBERTa) model, served via a spaCy v3.7.5 spancat pipeline. It extracts specific rhetorical engagement spans (e.g., ENTERTAIN, PROCLAIM, COUNTER) using a strict 0.05 confidence threshold.
- Data Compilation: The FastAPI routing layer merges the generative scorecard and the discriminative tag arrays into a unified JSON payload, guaranteeing zero data loss before returning the response to the client.
Full Pipeline
Results & Benchmarks
The system was benchmarked to ensure flawless data retention and network resiliency when executing heavy machine learning workloads on constrained edge hardware.
| Evaluation Metric | Benchmark Score | Significance |
|---|---|---|
| Payload Retention Rate | 100% | Achieved zero data-loss during API packaging by decoupling the sequence tagger from the LangGraph shared state memory. |
| Network Resiliency (Timeout Failures) | 0.0% | Bypassed strict 100-second cloud firewall limitations by implementing a persistent Ngrok TCP/HTTP tunnel bridging Vercel and the backend. |
| Extraction Confidence Threshold | 0.05 | Custom threshold applied to the PyTorch transformer model to guarantee granular detection of nuanced academic discourse markers. |
Findings
- Generative vs. Discriminative Limitations: Utilizing LLMs for exact span extraction proved highly unreliable due to tokenization shifts and generative hallucinations. The implementation of a dedicated discriminative NLP model (spaCy) alongside the LLM proved essential for academically rigorous analysis.
- Edge Inference Viability: Running memory-heavy PyTorch transformer weights on 1GB RAM instances is feasible when strategically leveraging hard drive swap memory and custom network tunneling to absorb the resulting inference latency.
Engineering Challenges & Architectural Solutions
Deploying a production-grade NLP pipeline within strict 100% free-tier resource constraints required resolving several critical engineering bottlenecks:
- The Hardware-Timeout Death Spiral: Google Cloud's free-tier `e2-micro` instances are strictly limited to 1GB of RAM. Loading a transformer model required paging memory to a slow hard drive swap, pushing API processing times to over 120 seconds. This triggered Cloudflare's hardcoded 100-second HTTP timeout limit, resulting in 524 Server Errors. This was resolved by abandoning Cloudflare and implementing an asynchronous Ngrok TCP/HTTP tunnel, allowing the connection to wait securely until inference was complete.
- State-Wiping Race Conditions in LangGraph: Running the generative multi-agent workflow and the custom NLP tag extraction within the same shared LangGraph `TypedDict` state caused the final aggregator agent to silently overwrite and delete the ML tags before JSON packaging. This was resolved by architecturally decoupling the system, running the spaCy model natively at the FastAPI routing layer, and merging the outputs externally.
- Canvas Rendering Failures: Exporting the dynamic evaluation scorecard using standard libraries (`html2canvas`, `jsPDF`) resulted in critical rendering failures, broken typography, and lost CSS grid data. External rendering bloat was stripped entirely and replaced with conditional Tailwind CSS visibility classes (`print:hidden`), hijacking the browser's native print engine to generate flawless PDF exports.
Limitations
- Inference Latency: Due to the intentional reliance on hard drive swap memory to prevent container OOM crashes on the 1GB RAM edge server, end-to-end evaluation latency is artificially high, averaging 1–2 minutes per request.
- Hardware Scaling Constraints: The current architecture requires manual `tmux` and Docker session configuration and does not support horizontal auto-scaling for concurrent user requests.