LLM System Optimization

You don't need a 2 trillion parameter model to tell you the capital of France is Paris. Be smart and route between a panel of models according to query difficulty and model specialty! New paper proposes a framework to train a router that routes queries to the appropriate LLM to optimize the trade-off b/w cost vs. performance. Overview: Model inference cost varies significantly: Per one million output tokens: Llama-3-70b ($1) vs. GPT-4-0613 ($60), Haiku ($1.25) vs. Opus ($75) The RouteLLM paper propose a router training framework based on human preference data and augmentation techniques, demonstrating over 2x cost saving on widely used benchmarks. They define the problem as having to choose between two classes of models: (1) strong models - produce high quality responses but at a high cost (GPT-4o, Claude3.5) (2) weak models - relatively lower quality and lower cost (Mixtral8x7B, Llama3-8b) A good router requires a deep understanding of the question’s complexity as well as the strengths and weaknesses of the available LLMs. Explore different routing approaches: - Similarity-weighted (SW) ranking - Matrix factorization - BERT query classifier - Causal LLM query classifier Neat Ideas to Build From: - Users can collect a small amount of in-domain data to improve performance for their specific use cases via dataset augmentation. - Can expand this problem from routing between a strong and weak LLM to a multiclass model routing approach where we have specialist models(language vision model, function calling model etc.) - Larger framework controlled by a router - imagine a system of 15-20 tuned small models and the router as the n+1'th model responsible for picking the LLM that will handle a particular query at inference time. - MoA architectures: Routing to different architectures of a Mixture of Agents would be a cool idea as well. Depending on the query you decide how many proposers there should be, how many layers in the mixture, what the aggregate models should be etc. - Route based caching: If you get redundant queries that are slightly different then route the query+previous answer to a small model to light rewriting instead of regenerating the answer

A candidate interviewing for Staff Engineer at Meta was asked to design an LLM router that sends 80% simple queries to a cheap 8B model and 20% harder ones to a 70B model without users noticing. Another candidate interviewing for L5 at Google was asked a very similar question. One person says: “I’ll classify the query as simple or hard. Simple queries go to the 8B model. Hard queries go to the 70B model.” Another person starts differently. They ask: - What does “simple” actually mean? - How confident is the router in that decision? - Which query categories should never go to the small model? - What happens when the 8B answer is low quality? - How do we measure whether users noticed the downgrade? One loses the offer. One gets hired. Btw, if you’re preparing for Senior to Principal-level system design interviews, I’ve put together 90+ fundamentals like this into a guide. You can check it out here: puneetpatwari.in This is only a high-level approach, but here is how I would break it down: [1] Define “simple” before routing anything A query is simple only when the intent is clear, the answer is factual, the risk is low, and the response does not need deep reasoning, tool use, personalization, or multi-step context. Examples: - password reset - basic FAQ - return policy - feature explanation - simple summarization Harder queries include: - billing disputes - legal or compliance questions - ambiguous user intent - multi-document reasoning - anything involving private account state The first move is not routing. The first move is classification with risk awareness. [2] Use confidence-based routing, not hard rules I would not build a binary router that blindly says “8B” or “70B.” I would create a routing layer that scores: - query complexity - intent clarity - domain risk - context required - user tier or business impact - historical success rate for similar queries If confidence is high, send it to the 8B model. If confidence is low, send it directly to 70B. If confidence is medium, use the 8B model first, but verify the answer before returning it. [3] Add a quality gate before the user sees the answer This is the part most candidates miss. The router should not only decide which model answers. It should also decide whether the answer is safe to show. For medium-risk queries, I would add: - lightweight judge model - factuality check against retrieved context - answer completeness check - fallback to 70B if confidence drops So the user does not feel the cheaper model. They only feel a fast, correct response. [4] Measure the router like a product system The real system will not be perfect on day one. So I would track: - cost per query - fallback rate - 8B success rate by category - user satisfaction - escalation to human support - answer correction rate - latency difference between routes Then I would roll it out gradually. Start with low-risk categories, prove quality, then expand coverage.

I’m jealous of AI Because with a model you can measure confidence Imagine you could do that as a human? Measure how close or far off you are? here's how to measure for technical and non-technical teams For business teams: Run a ‘known answers’ test. Give the model questions or tasks where you already know the answer. Think of it like a QA test for logic. If it can't pass here, it's not ready to run wild in your stack. Ask for confidence directly. Prompt it: “How sure are you about that answer on a scale of 1-10?” Then: “Why might this be wrong?” You'll surface uncertainty the model won't reveal unless asked. Check consistency. Phrase the same request five different ways. Is it giving stable answers? If not, revisit the product strategy for the llm Force reasoning. Use prompts like “Show step-by-step how you got this result.” This lets you audit the logic, not just the output. Great for strategy, legal, and product decisions. For technical teams: Use the softmax output to get predicted probabilities. Example: Model says “fraud” with 92% probability. Use entropy to spot uncertainty. High entropy = low confidence. (Shannon entropy: −∑p log p) Language models Extract token-level log-likelihoods from the model if you have API or model access. These give you the probability of each word generated. Use sequence likelihood to rank alternate responses. Common in RAG and search-ranking setups. For uncertainty estimates, try: Monte Carlo Dropout: Run the same input multiple times with dropout on. Compare outputs. High variance = low confidence. Ensemble models: Aggregate predictions from several models to smooth confidence. Calibration testing: Use a reliability diagram to check if predicted probabilities match actual outcomes. Use Expected Calibration Error (ECE) as a metric. Good models should show that 80% confident = ~80% correct. How to improve confidence (and make it trustworthy) Label smoothing during training Prevents overconfident predictions and improves generalization. Temperature tuning (post-hoc) Adjusts the softmax sharpness to better align confidence and accuracy. Temperature < 1 → sharper, more confident Temperature > 1 → more cautious, less spiky predictions Fine-tuning on domain-specific data Shrinks uncertainty and reduces hedging in model output. Especially effective for LLMs that need to be assertive in narrow domains (legal, medicine, strategy). Use focal loss for noisy or imbalanced datasets. It down-weights easy examples and forces the model to pay attention to harder cases, which tightens confidence on the edge cases. Reinforcement learning from human feedback (RLHF) Aligns the model's reward with correct and confident reasoning. Bottom line: A confident model isn't just better - it's safer, cheaper, and easier to debug. If you’re building workflows or products that rely on AI, but you’re not measuring model confidence, you’re guessing. #AI #ML #LLM #MachineLearning #AIConfidence #RLHF #ModelCalibration

I didn't ship a chatbot. I shipped a state machine. The prompt didn't fail. The memory routing did. Three weeks into building the vendor onboarding agent, I hit a hard wall. The flow looked clean in my notebook: ingest to extract to push to HubSpot. In production? It hallucinated compliance flags. Skipped signature blocks. Retried without context. I was treating the LLM like control flow. It's not. It's a stochastic function. You need explicit state. So I rebuilt it as a directed graph in LangGraph: 🔹 Ingest Node: PDF to chunking to semantic embedding to Pinecone for episodic memory storage 🔹 Extract Node: ReAct planner with Pydantic schema validation. If confidence drops below 0.82, route to fallback. 🔹 Validate Node: Guardrails AI plus regex plus business rule engine. Catches hallucinated tags before they hit the CRM. 🔹 Router Node: State-based conditional edges. New vendor triggers full flow. Existing vendor triggers delta update. Ambiguous input routes to HITL queue. The breakthrough wasn't better prompts. It was state management. I stopped asking the model to remember. I gave it a shared State object passed between nodes. I stopped hoping for accuracy. I set confidence thresholds, built semantic similarity checks, and routed failures to humans instead of guessing. I stopped tracking tokens. I tracked latency per node, cache hit rates, and fallback frequency. What changed in production: - Task completion: 62% to 89% - Hallucination rate: 0.3%, caught pre-CRM - HITL intervention: 38% to 12% - Cost per qualified record: $4.20 Agents aren't conversational UIs. They're state machines with stochastic components. If you're building GTM agents, stop optimizing for flow. Start optimizing for: ✅ Explicit state transitions, not chain-of-thought ✅ Confidence-based routing, not open-ended retries ✅ Observability traces via LangSmith or Phoenix, not open rates ✅ Human-in-the-loop as a fallback node, not an afterthought What's the hardest state transition you've had to engineer in your agent workflows? Drop your stack below. 👇 #AIGTM #LangGraph #MultiAgent #PMM #SystemDesign #AIEngineering #RAG #HITL #FieldNotes

💡 Why are we sending everything to expensive LLMs? The smarter architecture pattern I’m seeing work in the real world: ➡️ Route 70–90% of tasks to SLMs (Small Language Models) ➡️ Escalate only complex, ambiguous work to LLMs This isn’t just optimization—it’s necessary for scale. 🧠 Hybrid SLM + LLM Architecture (What Works) At the center is an AI Gateway that acts as the decision engine: - Classifies request complexity - Routes to the right model tier - Applies guardrails (PII, compliance) - Tracks cost + performance Execution model: - 🟢 SLMs → fast, cheap, high-volume tasks - 🔵 LLMs → deep reasoning, edge cases - 🔁 Fallback → escalate when confidence is low ⚡ Proven Patterns ✔️ SLM-first strategy (default routing) ✔️ Confidence-based escalation ✔️ Task decomposition (SLM → LLM chain) ✔️ RAG before generation ✔️ Aggressive caching ⚠️ Pitfalls I keep seeing ❌ Sending everything to LLMs → 💸 cost explosion ❌ Over-orchestrating “agentic” workflows → unnecessary complexity ❌ Ignoring latency → poor UX ❌ No cost observability → no control ❌ Same prompts for SLMs and LLMs → bad results 🧭 Simple mental model SLM = Worker LLM = Expert Let the workers handle the bulk. Call the expert only when it truly matters. 📊 What good looks like - 60–90% cost reduction - 2–5x faster response times - Better scalability without overengineering Most teams start with LLM-heavy designs. The winning approach is the opposite: 👉 Start small. Escalate to large. #AI #EnterpriseArchitecture #GenAI #AIArchitecture #CostOptimization #DigitalTransformation

Many “LLM routers” reduce to simple classifier heuristics, yet real-world routing demands handling cost, accuracy, and composition tradeoffs - a nuance many repos gloss over. LLMRouter brings structured routing to multi-LLM stacks by formalizing LLM selection as a decision problem over cost, performance, and task characteristics rather than a one-off API choice. The repository provides 16+ router implementations from classical baselines (KNN, SVM, MLP, Elo rating) to graph-based, multi-round, and personalized strategies, and integrates training, inference, and evaluation in a unified CLI with data pipelines from 11 benchmark datasets. Unlike toy classifiers, it embeds router training into an ML workflow with support for pre-trained multi-round routers such as Router-R1 (RL-trained policy router) and GMTRouter (graph-based personalization), surfacing concrete tradeoffs between simple heuristics and learned decision policies. Practically this elevates routing from hard-coded model selection to a reproducible engineering pattern. You get training data generation, metrics for performance vs cost, plugin hooks for custom logic, and API key driven inference pipelines; all together this reduces bespoke scripting and brittle ad-hoc logic that many teams build internally. The critical constraint remains operational overhead: router training and multi-round strategies add latency, GPU dependency for training, and complexity in monitoring cost/accuracy balance. In high-throughput production, this will require observability and failover design comparable to core inference layers. For AI architects evaluating multi-model stacks, LLMRouter is a substantive reference implementation showing how routing can be engineered and extended beyond simple task classification. Github👩💻https://lnkd.in/eJjFyAP5