Optimizing Technology Spending

You don't need a 2 trillion parameter model to tell you the capital of France is Paris. Be smart and route between a panel of models according to query difficulty and model specialty! New paper proposes a framework to train a router that routes queries to the appropriate LLM to optimize the trade-off b/w cost vs. performance. Overview: Model inference cost varies significantly: Per one million output tokens: Llama-3-70b ($1) vs. GPT-4-0613 ($60), Haiku ($1.25) vs. Opus ($75) The RouteLLM paper propose a router training framework based on human preference data and augmentation techniques, demonstrating over 2x cost saving on widely used benchmarks. They define the problem as having to choose between two classes of models: (1) strong models - produce high quality responses but at a high cost (GPT-4o, Claude3.5) (2) weak models - relatively lower quality and lower cost (Mixtral8x7B, Llama3-8b) A good router requires a deep understanding of the question’s complexity as well as the strengths and weaknesses of the available LLMs. Explore different routing approaches: - Similarity-weighted (SW) ranking - Matrix factorization - BERT query classifier - Causal LLM query classifier Neat Ideas to Build From: - Users can collect a small amount of in-domain data to improve performance for their specific use cases via dataset augmentation. - Can expand this problem from routing between a strong and weak LLM to a multiclass model routing approach where we have specialist models(language vision model, function calling model etc.) - Larger framework controlled by a router - imagine a system of 15-20 tuned small models and the router as the n+1'th model responsible for picking the LLM that will handle a particular query at inference time. - MoA architectures: Routing to different architectures of a Mixture of Agents would be a cool idea as well. Depending on the query you decide how many proposers there should be, how many layers in the mixture, what the aggregate models should be etc. - Route based caching: If you get redundant queries that are slightly different then route the query+previous answer to a small model to light rewriting instead of regenerating the answer

Over the last 18 months, the FinOps Foundation has seen a dramatic shift in the scope of spending that #FinOps practices manage beyond public cloud. We first explored this anticipated shift in the second edition of Cloud FinOps (pg. 401) where we shared a vision for how we expected the scope of FinOps to expand: to a world where FinOps practices are integrating costs beyond public cloud – from SaaS, to licensing, datacenter, and private cloud – for a more complete picture of cost to drive value-based decision-making across a broader scope of spending. In recent surveys, we are seeing upwards of 70% of practitioners now extending their practice beyond public cloud to other types of technology spend. To reflect this reality, the FinOps Foundation Technical Advisory Council has approved a new element in the FinOps Framework to capture the segments associated with the different types of technology cost and usage data FinOps Practitioners are managing: FinOps Scope. Read more in the new Insights article on the expanded scope of FinOps: https://lnkd.in/gPH3vQEn In some cases, especially for companies “born in the cloud,” FinOps teams are the only technology cost management team in the organization. In other cases, FinOps Practitioners are working alongside Allied Personas  (ITAM/ITSM/ITFM/TBM/SAM). But in all cases, FinOps’ success in managing cloud spending has the business asking “Can FinOps keep doing what you’re doing for cloud, AND also do it for X?” While other disciplines report on cost at a chargeback level, they do this for a monthly and quarterly roll-up of financial reporting at the general ledger level. FinOps, by contrast, is leveraging extremely granular cost and usage data at levels for all stakeholders, from engineering, to architecture, to product, to finance, and to executives, enabling them to: - Make information available outside of traditional silos to empower Personas across the organization, beyond Leadership – not just the CFO and CIO. - Enable timely decision-making about technology investment choices in “fixed” and variable Scopes. - Enable collaboration between technology and business teams at the engineering and product level. - Enable Cost Aware Product Decisions by bringing cost considerations earlier into the product development lifecycle. - Optimize, modernize, and automate to create consistency and iteratively improve technology usage and cost. Applying FinOps Capabilities to additional Scopes of spending gives businesses more comprehensive visibility into their technology costs. The goal for organizations is to understand and optimize the cost of offering each individual product or service. The first step is to get complete visibility into the cost of a product or service by pulling together all types of costs associated with delivering it... Read more in the new Insights article on the expanded scope of FinOps: https://lnkd.in/gPH3vQEn

If you’re an AI engineer trying to optimize your LLMs for inference, here’s a quick guide for you 👇 Efficient inference isn’t just about faster hardware, it’s a multi-layered design problem. From how you compress prompts to how your memory is managed across GPUs, everything impacts latency, throughput, and cost. Here’s a structured taxonomy of inference-time optimizations for LLMs: 1. Data-Level Optimization Reduce redundant tokens and unnecessary output computation. → Input Compression:  - Prompt Pruning, remove irrelevant history or system tokens  - Prompt Summarization, use model-generated summaries as input  - Soft Prompt Compression, encode static context using embeddings  - RAG, replace long prompts with retrieved documents plus compact queries → Output Organization:  - Pre-structure output to reduce decoding time and minimize sampling steps 2. Model-Level Optimization (a) Efficient Structure Design → Efficient FFN Design, use gated or sparsely-activated FFNs (e.g., SwiGLU) → Efficient Attention, FlashAttention, linear attention, or sliding window for long context → Transformer Alternates, e.g., Mamba, Reformer for memory-efficient decoding → Multi/Group-Query Attention, share keys/values across heads to reduce KV cache size → Low-Complexity Attention, replace full softmax with approximations (e.g., Linformer) (b) Model Compression → Quantization:  - Post-Training, no retraining needed  - Quantization-Aware Training, better accuracy, especially <8-bit → Sparsification:  - Weight Pruning, Sparse Attention → Structure Optimization:  - Neural Architecture Search, Structure Factorization → Knowledge Distillation:  - White-box, student learns internal states  - Black-box, student mimics output logits → Dynamic Inference, adaptive early exits or skipping blocks based on input complexity 3. System-Level Optimization (a) Inference Engine → Graph & Operator Optimization, use ONNX, TensorRT, BetterTransformer for op fusion → Speculative Decoding, use a smaller model to draft tokens, validate with full model → Memory Management, KV cache reuse, paging strategies (e.g., PagedAttention in vLLM) (b) Serving System → Batching, group requests with similar lengths for throughput gains → Scheduling, token-level preemption (e.g., TGI, vLLM schedulers) → Distributed Systems, use tensor, pipeline, or model parallelism to scale across GPUs My Two Cents 🫰 → Always benchmark end-to-end latency, not just token decode speed → For production, 8-bit or 4-bit quantized models with MQA and PagedAttention give the best price/performance → If using long context (>64k), consider sliding attention plus RAG, not full dense memory → Use speculative decoding and batching for chat applications with high concurrency → LLM inference is a systems problem. Optimizing it requires thinking holistically, from tokens to tensors to threads. Image inspo: A Survey on Efficient Inference for Large Language Models ---- Follow me (Aishwarya Srinivasan) for more AI insights!

𝗪𝗵𝘆 𝗧𝗕𝗠 𝗶𝘀 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 𝘂𝗻𝗱𝗲𝗿𝗿𝗮𝘁𝗲𝗱 𝗰𝗼𝘀𝘁 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆? Everyone talks about FinOps when it comes to cloud cost control. But TBM? It’s the only framework that provides a structured way to align IT spending - both digital and non-digital - with business value. Today most IT cost-cutting efforts focus on cloud costs. But what about on-prem data centers, networking, end-user computing, software licensing, IT service management, and physical infrastructure? That’s where TBM shines. Unlike FinOps, which primarily focuses on cloud cost management, TBM covers all IT spend - digital and non-digital. That means: ✓ On-prem data centers (server costs, cooling, power, maintenance) ✓ SaaS and enterprise software (license costs, renewals, shadow IT) ✓ Network infrastructure (bandwidth costs, MPLS, SD-WAN optimizations) ✓ End-user computing (desktops, mobile devices, IT support costs) ✓ IT services & outsourcing (managed services, BPOs, contract negotiations) This is what makes TBM different - it breaks IT costs into layers: ✓ Cost Pools – The raw IT expenses (hardware, software, labor, facilities, etc.). ✓ IT Towers – Logical groupings like compute, storage, network, and applications. ✓ Products & Services – The services IT delivers (e.g., CRM platforms, cloud storage, collaboration tools). ✓ Business Units – The actual consumers of IT resources (sales, marketing, HR, etc.). This multi-layer mapping gives granular visibility into IT spending. This enables CIOs and CFOs optimize across hybrid IT environments. 𝗪𝗵𝘆 𝗜 𝗹𝗼𝘃𝗲 𝗧𝗕𝗠? Most organizations optimize reactively - shutting down workloads, cutting headcount, or delaying upgrades. TBM forces a proactive, data-driven approach by integrating: ✓ Cost transparency – Mapping IT costs to business units, services, and outcomes ✓ Showback/chargeback – Assigning costs directly to business teams for accountability ✓ Unit economics – Measuring IT efficiency per unit of business value (cost per transaction, cost per API call, etc.) ✓ Benchmarking – Comparing internal IT costs with industry standards to identify waste The result? ✓ IT isn’t just seen as a cost center - it becomes a strategic partner. ✓ Cost-cutting doesn’t compromise performance or innovation. ✓ Businesses make smarter investment decisions, balancing cost, quality, and value. Why TBM is still underappreciated? TBM doesn’t promise quick fixes. It requires a mature cost culture, strong leadership, and deep integration into financial planning. And the truth is - many companies don’t want to do the hard work. They’d rather cut budgets blindly than ask the harder question: "Is this IT spend actually driving business value?" The companies that do embrace TBM gain full control over IT costs - cloud, data center, software, infrastructure, services, everything. TBM is about spending right, not spending less. #TBM Technology Business Management (TBM) Council

IDC links the Middle East conflict to global IT spending. Energy prices drive costs across data centers, semiconductor fabrication, cloud pricing, and logistics. Every link in the AI infrastructure chain runs on power. When oil moves, your deployment costs move with it. Their baseline scenario assumes a contained conflict. Global IT spending is still growing near 10% in 2026. But if escalation extends to three months, growth drops by a full percentage point. Device budgets and discretionary projects absorb the hit first. Memory chip supply was already tight before the conflict. Defense demand for advanced semiconductors could further squeeze availability and push DRAM and NAND pricing higher. That flows straight into AI hardware and enterprise storage costs. Three things IDC says stay funded regardless of how bad it gets: AI infrastructure, because it's embedded in the core strategy, and productivity gains matter more during inflation. Cybersecurity, because geopolitical escalation drives state-sponsored attacks, and nobody cuts security spending during a conflict. Sovereign digital platforms, because governments in the Gulf are accelerating national control over cloud and computing Everything else faces reprioritization. Consumer tech, device refresh cycles, and nonessential enterprise projects slow down first. Most IT budgets I review are built on the assumption of stable energy prices and predictable hardware costs. Neither assumption holds right now. If your AI deployment plan doesn't include a scenario where oil stays above $100 for six months, you're planning on conditions that may not exist by Q3. Stress-test the budget before the market does it for you. #Geopolitics #EnterpriseAI #ITSpending #AIInfrastructure #Cybersecurity #MiddleEast #Semiconductors #CIO #CFO #BusinessStrategy #DigitalTransformation #RiskManagement

AI adoption and usage is skyrocketing, but usage bills are running up fast. The good news? There’s a bunch of ways to cut costs without sacrificing quality. Here are ten practical ways to optimize your (and your team’s) AI spend: 1. Right model, right task - You don't need a frontier model to summarize an email or parse a PDF. Routing simple tasks to smaller models can cut costs by 40% to 85% with a negligible drop in quality. Different models are better at different tasks. 2. Mind the token asymmetry - Output tokens are usually 3x to 4x more expensive than input tokens. Be explicit about expected outputs - asking for concise summaries, structured JSON or a specific output format can cut down costs and rework. 3. Pre-process first - Don't waste expensive model context windows on raw formatting noise eg convert PDFs or PowerPoints to Markdown using standard software before involving an LLM. This is especially important for very large data sets or documents. 4. Use single-turn alignment - Back and forth chat resends the whole conversation history with every prompt, compounding input costs. Try to put all your requirements, constraints, and output formats in one clear initial prompt. 5. Maximize token allowances - Schedule non-urgent batch tasks for off-peak hours or near the end of a billing cycle to use up existing capacity. Share unused tokens across team members. 6. Clean up inactive seats - Research shows 30% to 50% of enterprise SaaS licenses go underutilised. Regularly audit user accounts, reclaim seats from inactive staff, and right-size plans before renewals. 7. Track usage - You can't manage what you don't measure. Implement basic monitoring to spot runaway queries early, and identify high-value internal use cases. 8. Share the learning - Trial-and-error across a team burns through API budgets fast. Share best practices and firm wide knowledge, build internal prompt libraries to share best practices so employees aren't reinventing the wheel. 9. Have options - Have a backup in case one tool hits an unexpected bottleneck. Some providers let you carry forward unused tokens across billing cycles, and there are some free options too. 10. Ask for help - You can ask some models to help you plan an activity and get it to spin up cheaper subagents to keep the cost down and quality right. Managing AI costs isn't about being cheap - it's about being smart and operational maturity. If the model providers do keep increasing the cost of their frontier model then we’ll all have to be smarter in how we use their tools. What is your team’s favorite strategy for keeping AI costs under control?

𝗔𝗿𝗲 𝘆𝗼𝘂 𝗽𝗿𝗼𝗮𝗰𝘁𝗶𝘃𝗲𝗹𝘆 𝗺𝗮𝗻𝗮𝗴𝗶𝗻𝗴 𝘆𝗼𝘂𝗿 𝗦𝗼𝘂𝗿𝗰𝗲-𝘁𝗼-𝗣𝗮𝘆 𝘁𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝘆 𝗰𝗼𝘀𝘁𝘀? If not, why let savings from smart Procurement slip away due to outdated technology or suboptimal use? S2P technology plays a central role in cost management, yet many companies lack a strategic approach to continuously assess and optimise their tech stack. Companies can adopt Bain & Co’s "𝗥𝗲𝗱𝘂𝗰𝗲, 𝗥𝗲𝗽𝗹𝗮𝗰𝗲, 𝗮𝗻𝗱 𝗥𝗲𝘁𝗵𝗶𝗻𝗸" model to continuously evaluate their technology infrastructure and costs, ensuring a more optimised and sustainable cost profile. Here is the model in action for Source to Pay technology cost optimisation: ▪️ 𝗥𝗲𝗱𝘂𝗰𝗲 to recover 10 to 20% of costs through short-term actions such as - adjusting licenses to match actual usage and adoption patterns - discontinuing features or functionalities that add little value - switching off modules where business capabilities have not yet caught up Avoid over-licensing by matching user access to actual needs, ensuring modules align with Procurement’s readiness. ▪️ 𝗥𝗲𝗽𝗹𝗮𝗰𝗲 to yield 20 to 30% of savings by - transitioning to cost-optimal, flexible solutions and getting out of lock-ins - switching subscription models when premium offerings are unnecessary - consolidating overlapping tools that offer similar features For example, merge multiple eSourcing tools into a primary platform and adopt a tender-based pricing for niche auction needs. This helps to adjust the cost profile of your Source to Pay technology with the actual needs. ▪️ 𝗥𝗲𝘁𝗵𝗶𝗻𝗸 to realise up to 40% cost optimisation by: - reimagining the architecture with a modular, composable design - automating and orchestrating processes and integrating new digital tools - reevaluate the mix of best-of-breed solutions vs integrated suites A new Procurement strategy requires a fresh look at the S2P tech stack to ensure it adapts and supports growth cost-effectively, while offering flexibility through additional digital levers like AI and automation. 𝗢𝗽𝘁𝗶𝗺𝗶𝘀𝗶𝗻𝗴 𝗦𝟮𝗣 𝘁𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝘆 𝗶𝘀 𝗮 𝗰𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗷𝗼𝘂𝗿𝗻𝗲𝘆, 𝗻𝗼𝘁 𝗮 𝗼𝗻𝗲-𝘁𝗶𝗺𝗲 𝗲𝗳𝗳𝗼𝗿𝘁, especially with contractual commitments, sunk costs, and change management challenges. Rather than following IT preferences and standards, it’s about keeping technology fresh and aligned with business needs as they evolve. ❓How do you manage your S2P technology to adapt to changing business needs while maintaining cost efficiency.

After optimizing costs for many AI systems, I've developed a systematic approach that consistently delivers cost reductions of 60-80%. Here's my playbook, in order of least to most effort: Step 1: Optimizing Inference Throughput Start here for the biggest wins with least effort. Enabling caching (LiteLLM (YC W23), Zilliz) and strategic batch processing can reduce costs by a lot with very little effort. I have seen teams cut costs by half simply by implementing caching and batching requests that don't require real-time results. Step 2: Maximizing Token Efficiency This can give you an additional 50% cost savings. Prompt engineering, automated compression (ScaleDown), and structured outputs can cut token usage without sacrificing quality. Small changes in how you craft prompts can lead to massive savings at scale. Step 3: Model Orchestration Use routers and cascades to send prompts to the cheapest and most effective model for that prompt (OpenRouter, Martian). Why use GPT-4 for simple classification when GPT-3.5 will do? Smart routing ensures you're not overpaying for intelligence you don't need. Step 4: Self-Hosting I only suggest self-hosting for teams at scale because of the complexities involved. This requires more technical investment upfront but pays dividends for high-volume applications. The key is tackling these layers systematically. Most teams jump straight to self-hosting or model switching, but the real savings come from optimizing throughput and token efficiency first. What's your experience with AI cost optimization?

We invested a significant amount on technology, so we defined 6 principles to make sure that investment translates into lasting impact: 1. Prioritize for Scale: Solve significant, enterprisewide challenges using solutions designed to scale globally, avoiding isolated or short-term fixes that don’t translate broadly. 2. Think in Horizons: Balance short-term operational necessities (Horizon 1: immediate wins within 6 to 12 months) with strategic planning for mid-to-long-term innovation (Horizons 2 & 3: 3 to 5 years ahead). 3. Data-Driven Decisions: Use independent, fact-based assessments and clear data, not subjective opinions or transient trends to guide investment choices and technology priorities. 4. Partnership First (Balance Buy vs Build): Strategically choose partnerships or external technology providers for speed and differentiation, ensuring faster time-to-market and competitive advantage, while balancing strategic internal development. 5. Dynamic Allocation: Frequently reassess and realign investments proactively, responding swiftly to rapid technological evolution and emerging opportunities. 6. Unity through Clarity: Clearly communicate the implications of technology decisions on infrastructure, talent, suppliers, and existing initiatives, ensuring stakeholder alignment and continuous support. We learned a lot by looking inward. If this helps you avoid even one wrong turn - I’m glad we shared it.

In a recent roundtable with fellow CXOs, a recurring theme emerged: the staggering costs associated with artificial intelligence (AI) implementation. While AI promises transformative benefits, many organizations find themselves grappling with unexpectedly high Total Cost of Ownership (TCO). Businesses are seeking innovative ways to optimize AI spending without compromising performance. Two pain points stood out in our discussion: module customization and production-readiness costs. AI isn't just about implementation; it's about sustainable integration. The real challenge lies in making AI cost-effective throughout its lifecycle. The real value of AI is not in the model, but in the data and infrastructure that supports it. As AI becomes increasingly essential for competitive advantage, how can businesses optimize costs to make it more accessible? Strategies for AI Cost Optimization 1.Efficient Customization - Leverage low-code/no-code platforms can reduce development time - Utilize pre-trained models and transfer learning to cut down on customization needs 2. Streamlined Production Deployment - Implement MLOps practices for faster time-to-market for AI projects - Adopt containerization and orchestration tools to improve resource utilization 3. Cloud Cost Management -Use spot instances and auto-scaling to reduce cloud costs for non-critical workloads. - Leverage reserved instances For predictable, long-term usage. These savings can reach good dollars compared to on-demand pricing. 4.Hardware Optimization - Implement edge computing to reduce data transfer costs - Invest in specialized AI chips that can offer better performance per watt compared to general-purpose processors. 5.Software Efficiency - Right LLMS for all queries rather than single big LLM is being tried by many - Apply model compression techniques such as Pruning and quantization that can reduce model size without significant accuracy loss. - Adopt efficient training algorithms Techniques like mixed precision training to speed up the process -By streamlining repetitive tasks, organizations can reallocate resources to more strategic initiatives 6.Data Optimization - Focus on data quality since it can reduce training iterations - Utilize synthetic data to supplement expensive real-world data, potentially cutting data acquisition costs. In conclusion, embracing AI-driven strategies for cost optimization is not just a trend; it is a necessity for organizations looking to thrive in today's competitive landscape. By leveraging AI, businesses can not only optimize their costs but also enhance their operational efficiency, paving the way for sustainable growth. What other AI cost optimization strategies have you found effective? Share your insights below! #MachineLearning #DataScience #CostEfficiency #Business #Technology #Innovation #ganitinc #AIOptimization #CostEfficiency #EnterpriseAI #TechInnovation #AITCO