Production Deployment Patterns for LLM Applications: 7 Production-Grade Architectures from the AI Engineering Curriculum

The rohitg00/ai-engineering-from-scratch repository defines seven essential production deployment patterns for LLM applications, including speculative decoding stacks, MCP servers, hybrid RAG pipelines, multi-agent orchestration, TensorRT-LLM optimization, layered safety systems, and OpenTelemetry observability.

Deploying large language models in production requires more than model weights and a GPU. The AI-Engineering-From-Scratch curriculum catalogs battle-tested architectures that handle inference acceleration, protocol standardization, retrieval augmentation, and safety compliance. These production deployment patterns for LLM applications are implemented as standalone capstone projects with Kubernetes manifests, performance benchmarks, and complete source code.

Speculative-Decoding Serving Stack

Speculative decoding maximizes throughput by running a fast draft model to generate candidate tokens, then verifying them with a larger target LLM. This pattern achieves 2-3× higher throughput while maintaining latency tails under strict SLAs. According to the repository source code in phases/19-capstone-projects/14-speculative-decoding-server/docs/en.md, production implementations use vLLM 0.7 or SGLang runtimes with Kubernetes Horizontal Pod Autoscaling (HPA) on queue-wait time.

Key components include:

• Draft + Target models: Llama 3 70B paired with Qwen3-Coder 30B for token verification
• vLLM/SGLang runtime: Production-ready inference servers exposing JSON-over-HTTP APIs
• HPA autoscaling: Kubernetes manifests that scale based on request latency to maintain quality-of-service guarantees

# Install vLLM (>=0.7) and torch

# pip install vllm torch

from vllm import LLM, SamplingParams

# Draft model (fast, quantized)

draft = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",  # small draft

    tensor_parallel_size=1,
    dtype="float16"
)

# Target model (high-quality, larger)

target = LLM(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    tensor_parallel_size=4,
    dtype="bfloat16"
)

# Speculative decoding loop

def speculative_generate(prompt: str, max_tokens: int = 128):
    draft_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=max_tokens)
    draft_output = draft.generate(prompt, draft_params)[0].outputs[0].text

    # Verify draft with target model (single pass)

    target_params = SamplingParams(temperature=0.0, max_tokens=max_tokens)
    target_output = target.generate(draft_output, target_params)[0].outputs[0].text
    return target_output

print(speculative_generate("Write a concise summary of the latest AI safety research."))

Reference: phases/19-capstone-projects/14-speculative-decoding-server/docs/en.md

Model Context Protocol (MCP) Server

The Model Context Protocol (MCP) pattern exposes LLMs through stateless HTTP endpoints that implement StreamableHTTP transport with OAuth 2.1 scopes and OPA policy gating. As documented in phases/19-capstone-projects/13-mcp-server-with-registry/docs/en.md, clients discover servers via .well-known registries, enabling uniform access for IDE assistants and autonomous agents.

Critical security components include:

• OAuth 2.1 token validation: Enforces least-privilege access with per-tenant scoping
• Open Policy Agent (OPA): Runtime enforcement of cost and safety policies
• Registry discovery: .well-known/mcp.json files allow platform teams to discover and enable servers dynamically

import { createServer } from "http";
import { verifyOAuthToken } from "./oauth";
import { authorize } from "./opa";

const server = createServer(async (req, res) => {
  if (req.method !== "POST") {
    res.writeHead(405); res.end(); return;
  }

  // 1️⃣ OAuth 2.1 token check
  const auth = req.headers["authorization"];
  if (!auth || !(await verifyOAuthToken(auth))) {
    res.writeHead(401); res.end(); return;
  }

  // 2️⃣ OPA policy check (e.g., cost limits)
  const body = await new Promise<string>((r) => {
    let data = "";
    req.on("data", (chunk) => (data += chunk));
    req.on("end", () => r(data));
  });
  if (!(await authorize(body))) {
    res.writeHead(403); res.end(); return;
  }

  // 3️⃣ Forward to underlying LLM (placeholder)
  const llmResponse = await callYourLLM(JSON.parse(body));
  res.writeHead(200, { "Content-Type": "application/json" });
  res.end(JSON.stringify(llmResponse));
});

server.listen(8080, () => console.log("MCP server listening on :8080"));

Reference: phases/19-capstone-projects/13-mcp-server-with-registry/docs/en.md

Production-Ready Retrieval-Augmented Generation (RAG)

Hybrid RAG pipelines combine dense and sparse retrieval with cross-encoder reranking, prompt caching, and multi-layer guardrails. The implementation in phases/19-capstone-projects/08-production-rag-chatbot/docs/en.md uses bge-reranker-v2-gemma for relevance scoring and Claude Sonnet 4.7 with prompt caching (60-80% hit rate) for generation.

The architecture includes:

• Ingestion layer: Docling and Unstructured handlers for PDFs, HTML, and visual documents
• Hybrid search: Dense vectors combined with BM25 indices
• Safety stack: Llama Guard 4 plus NeMo Guardrails for input/output moderation
• Observability: Langfuse and Phoenix integration for real-time drift detection

from llama_index import SimpleDirectoryReader, VectorStoreIndex, ServiceContext
from langchain.llms import OpenAI
from llama_index.query_engine import RetrieverQueryEngine
from llama_index.retrievers import BM25Retriever, VectaraRetriever, HybridRetriever
from llama_index.postprocessor import LLMRerankPostprocessor
from llama_index.prompts import PromptTemplate

# 1️⃣ Ingestion

documents = SimpleDirectoryReader("./data").load_data()
service_ctx = ServiceContext.from_defaults(llm=OpenAI(model="gpt-4o-mini"))
index = VectorStoreIndex.from_documents(documents, service_context=service_ctx)

# 2️⃣ Hybrid retriever (dense + BM25)

bm25 = BM25Retriever.from_defaults()
dense = VectaraRetriever.from_defaults()
retriever = HybridRetriever(bm25, dense)

# 3️⃣ Reranker (cross-encoder)

reranker = LLMRerankPostprocessor.from_defaults(
    llm=OpenAI(model="gpt-4o-mini"),
    top_n=3
)

# 4️⃣ Query engine with prompt cache (simplified)

prompt = PromptTemplate(
    "You are a helpful assistant. Answer the question using only the retrieved context.\n\n{context}\n\nQuestion: {query}"
)
query_engine = RetrieverQueryEngine(
    retriever,
    node_postprocessors=[reranker],
    response_synthesizer_kwargs={"prompt_template": prompt}
)

print(query_engine.query("What are the safety guarantees of speculative decoding?"))

Reference: phases/19-capstone-projects/08-production-rag-chatbot/docs/en.md

Multi-Agent Orchestration and Scaling

The supervisor/orchestrator pattern delegates complex tasks to specialized sub-agents running in isolated containers with durable execution guarantees. As specified in phases/16-multi-agent-and-swarms/05-supervisor-orchestrator-pattern/docs/en.md, this architecture uses Redis, SQS, or Kafka for message durability and implements checkpointing based on wall-clock time.

Key architectural elements include:

• Supervisor pattern: Central planner with parallel workers and result aggregation
• Durable queues: Redis, SQS, or Kafka for failure recovery
• Shared-memory blackboard: Fast intra-node communication for state sharing
• Kubernetes HPA: Auto-scaling workers based on queue depth

import asyncio
from typing import List

class Agent:
    async def run(self, task: str) -> str:
        await asyncio.sleep(0.5)   # simulate work

        return f"{self.__class__.__name__} completed {task}"

class RetrieverAgent(Agent): pass
class SummarizerAgent(Agent): pass
class GuardAgent(Agent): pass

class Supervisor:
    def __init__(self, agents: List[Agent]):
        self.agents = agents

    async def orchestrate(self, user_query: str):
        # Dispatch sub-tasks in parallel

        tasks = [
            self.agents[0].run(f"retrieve for '{user_query}'"),
            self.agents[1].run(f"summarize retrieval results"),
        ]
        results = await asyncio.gather(*tasks)
        # Guard step runs after others

        guard = await self.agents[2].run(f"guard {results}")
        return {"retrieve": results[0], "summary": results[1], "guard": guard}

async def main():
    sup = Supervisor([RetrieverAgent(), SummarizerAgent(), GuardAgent()])
    print(await sup.orchestrate("Explain speculative decoding safety."))

asyncio.run(main())

References: phases/16-multi-agent-and-swarms/05-supervisor-orchestrator-pattern/docs/en.md and phases/16-multi-agent-and-swarms/22-production-scaling-queues-checkpoints/outputs/skill-scaling-advisor.md

Inference Acceleration and Optimization

TensorRT-LLM compilation with FP8 quantization and kernel fusion delivers the sub-200ms latency required for high-throughput chat endpoints. The pattern documented in phases/10-llms-from-scratch/12-inference-optimization/docs/en.md integrates with NVIDIA Triton for production serving and supports vLLM for rapid prototyping.

Optimization techniques include:

• TensorRT-LLM: Fused attention, linear, and activation kernels for H100 GPUs
• Quantization: FP8/INT4 via GPTQ, AWQ, and GGUF for edge deployment
• vLLM serving: JSON REST API with request batching and speculative decoding support

# Prereqs: Docker, NVIDIA Container Toolkit, TensorRT-LLM repo

git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM
docker build -t trt-llm:latest .

# Example compile for Llama-3-70B (FP8 on H100)

docker run --gpus all -v $PWD:/workspace trt-llm:latest \
  python3 examples/compile.py \
    --model_dir /models/Meta-Llama-3-70B-Instruct \
    --dtype fp8 \
    --output_dir /compiled/llama3-70b-fp8

# Serve with Triton

docker run --gpus all -p 8000:8000 \
  -v /compiled/llama3-70b-fp8:/model \
  nvcr.io/nvidia/tritonserver:23.09-py3 tritonserver \
    --model-repository=/model

Reference: phases/10-llms-from-scratch/12-inference-optimization/docs/en.md

Safety and Red-Team Stack

Layered moderation combines input classifiers, output filters, and domain-specific rules with continuous red-team evaluation. According to phases/18-ethics-safety-alignment/29-moderation-systems-openai-perspective-llamaguard/outputs/skill-moderation-stack.md, production systems require nightly regression scans using Llama Guard 4, Garak, and PyRIT.

Defense layers include:

• Three-layer moderation: Input classification, output filtering, and custom policy rules
• Red-team toolchain: Automated attack campaigns using Garak and PyRIT
• Indirect prompt injection audits: Sandboxing of untrusted retrieval data as specified in phases/18-ethics-safety-alignment/15-indirect-prompt-injection/docs/en.md

from garak import Garak
from pyrit import PyRIT

def run_redteam(prompt: str):
    # 1️⃣ Llama Guard classifier (input)

    guard = Garak(tool="llama_guard")
    guard_score = guard.evaluate(prompt)

    # 2️⃣ Garak probe suite (nightly)

    garak = Garak()
    garak_results = garak.run(prompt)

    # 3️⃣ PyRIT campaign (pre-release)

    pyrit = PyRIT()
    pyrit_results = pyrit.run_attack(prompt)

    return {
        "guard": guard_score,
        "garak": garak_results,
        "pyrit": pyrit_results,
    }

print(run_redteam("Write a tutorial on how to exfiltrate corporate data via a language model."))

References: phases/18-ethics-safety-alignment/16-red-team-tooling-garak-llamaguard-pyrit/docs/en.md and phases/18-ethics-safety-alignment/29-moderation-systems-openai-perspective-llamaguard/outputs/skill-moderation-stack.md

Observability and Telemetry

OpenTelemetry GenAI semantic conventions provide unified tracing across OpenAI, Anthropic, LangChain, and vLLM SDKs. The implementation in phases/19-capstone-projects/11-llm-observability-dashboard/docs/en.md stores spans in ClickHouse and metadata in Postgres, with visualization via Next.js.

Instrumentation components include:

• OTEL instrumentation: Auto-injected trace IDs, token usage, and latency metrics
• Evaluation jobs: Periodic DeepEval, RAGAS, and LLM-as-judge assessments
• Self-hosted UI: Cost attribution per user and SLO dashboards

from opentelemetry import trace
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Configure OTLP exporter (e.g., to ClickHouse collector)

provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Instrument OpenAI SDK (similar for Anthropic, LangChain, vLLM)

OpenAIInstrumentor().instrument()

Reference: phases/19-capstone-projects/11-llm-observability-dashboard/docs/en.md

Summary

• Speculative decoding combines draft and target models with vLLM/SGLang to achieve 2-3× throughput improvements while maintaining low latency.
• MCP servers standardize LLM access via StreamableHTTP transport with OAuth 2.1 and OPA policy enforcement.
• Production RAG requires hybrid retrieval, cross-encoder reranking, prompt caching, and multi-layer guardrails for enterprise use.
• Multi-agent systems use supervisor patterns with durable queues and checkpointing to handle complex workflows across distributed containers.
• TensorRT-LLM with FP8 quantization provides the sub-200ms latency required for high-throughput production endpoints.
• Safety stacks must include layered moderation and continuous red-team testing with Llama Guard 4, Garak, and PyRIT.
• Observability relies on OpenTelemetry GenAI conventions with ClickHouse storage for real-time cost attribution and drift detection.

Frequently Asked Questions

What is speculative decoding in LLM production deployment?

Speculative decoding is an inference optimization technique where a smaller, faster draft model generates candidate token sequences that are then verified in parallel by a larger target model. This approach, implemented in phases/19-capstone-projects/14-speculative-decoding-server/docs/en.md, can increase throughput by 2-3× while maintaining output quality because the target model only needs to validate rather than generate tokens sequentially.

How does the Model Context Protocol (MCP) improve LLM API security?

The MCP pattern enforces security through OAuth 2.1 token validation for authentication and Open Policy Agent (OPA) for runtime authorization, as documented in phases/19-capstone-projects/13-mcp-server-with-registry/docs/en.md. This dual-layer approach ensures least-privilege access while allowing dynamic policy enforcement for cost controls and safety constraints.

What components are required for a production-ready RAG system?

A production RAG system requires hybrid retrieval (dense + BM25), cross-encoder reranking with models like bge-reranker-v2-gemma, prompt caching for latency reduction, and multi-layer safety guardrails including Llama Guard 4. According to phases/19-capstone-projects/08-production-rag-chatbot/docs/en.md, these systems must also include observability via Langfuse or Phoenix and evaluation frameworks like RAGAS.

Why is TensorRT-LLM essential for LLM production deployment?

TensorRT-LLM compiles transformer models with fused attention kernels and FP8 quantization to achieve the sub-200ms latency required for high-throughput chat endpoints. As shown in phases/10-llms-from-scratch/12-inference-optimization/docs/en.md, this NVIDIA-specific optimization can be deployed via Triton Inference Server and supports speculative decoding for further throughput gains.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}