Gabriele Iuculano

MoEs are great. They let you increase the parameter count of the model without increasing the compute. But they also not free. Here are 3 challenges of having MoEs in your models. 1. If you have a big model, you probably would think about partitioning the model and then collate the results. But in models with MoEs the number of active experts K << total number of experts E. This means for any particular token the amount of computation is very less. If you just do any of the normal sharding methods such as tensor parallelism, already having small computations would make the individual computations even smaller and so most of the time your GPU is going to sit idle. Solution is to have Expert Parallelism, which means different Experts on different GPUs. But this also means better communication demands and we need to be careful about how we are routing the different tokens to the different experts. 2. Only K activate per token, but we need all the E experts' parameters, gradients and optimizer states to be there in memory. Its like you have work for 5 people, but you hire 50; surely they will one day revolt and overthrow you. Jokes aside, you can see that this is obviously quite wasteful and creates a lot of register pressure. If we want to reduce this pressure this means we have to do something else. Like distribute more parameters which means more communication, or maybe recompute some activations which means more FLOPs, or offloading to CPU host memory will mean your model runs slower. 3. Routing and communication overhead. You need to route the tokens efficiently. During training you need to worry about if all the models are trained equally and make sure that only some of the experts should not be hoarding majority of the tokens. Expert parallelism means that all to all collectives are required to dispatch tokens to their assigned experts and then once done collect the results. This means a huge amount data transfer per GPU per step. This means that we should be aiming to limit the amount of parallelism as much as possible which requires careful planning and optimisation. Source: https://lnkd.in/gw4KpENa

30

Why Markdown (.md) Files Are a Hidden Superpower in LLM Systems Everyone is optimizing models. Very few are optimizing inputs. After building large-scale RAG and agentic AI systems, one architectural decision consistently improves performance: Convert everything to Markdown before feeding it to an LLM. Here’s why: 1-Clean Semantic Structure Markdown provides hierarchy without noise: Ex- # Main Topic ## Subtopic - Key point No XML (DOCX), no layout artifacts (PDF), no script clutter (HTML). LLMs process tokens and patterns. Markdown gives structure with minimal token waste. 2-Better RAG Chunking With Markdown, you can chunk by headings instead of arbitrary token splits. That means: • Section-aware embeddings • Higher retrieval precision • Fewer hallucinations Frameworks like LangChain and LlamaIndex encourage structured documents for this reason. 3-Token Efficiency = More Reasoning Space <h2 style="color:red">Termination</h2> vs ## Termination Less formatting overhead → more usable context window. Critical when using models from OpenAI and other frontier providers. 4-Real Example: OpenClaw OpenClaw structures its system using modular .md files like: • soul.md • skills.md • traits.md Each concept is isolated, hierarchical, and retrievable. That’s essentially RAG-native design. Markdown becomes a lightweight knowledge graph. The Core Insight LLMs are pattern learners. Markdown mirrors human thinking: Headings → Sections → Lists → Blocks That alignment improves: ✔ Retrieval ✔ Reasoning ✔ Maintainability ✔ Cost efficiency Bigger models help. Cleaner inputs help more. If you’re building RAG or AI copilots — standardize your knowledge base in Markdown first.

11

What’s the Deal with Karpathy’s “microGPT” – and Why Is Everyone Talking About It? microGPT is not a new super-model, but a deliberately tiny GPT clone: a complete Transformer in about 200 lines of pure Python, without PyTorch, without NumPy, without GPU. It learns, for example, from a list of about 32,000 first names to then “hallucinate” new, plausible-sounding names. What’s exciting is not what the model can do, but how radically Karpathy simplifies it: One script contains everything: dataset, tokenizer, autograd, GPT-like architecture, Adam optimizer, training and inference loop.​ About 4,192 parameters, 16-dimensional embeddings, 4 attention heads, 1 layer – a toy model, but architecturally closely related to large LLMs.​ All based on simple Python scalars and a hand-built autograd mechanism.​ Why do many find it “groundbreaking”? It strips away the black-box magic: Anyone who understands microGPT grasps the core of what happens in ChatGPT & Co. – just in miniature format. It massively lowers the entry barrier: Instead of millions of lines of production code, you get a single, readable file that you can follow and replicate line by line.​ In short: microGPT is not a step toward “even bigger” models, but a step toward more understanding. It shows that behind modern LLMs lies a surprisingly compact algorithm – and that every developer can comprehend this algorithm. I love ❤️ it and will of course port it to pure CUDA :) Try yourself: https://lnkd.in/drdQux58

38

1 Kommentar

Couldn’t help myself — had to share this part of the Andrew Ng conversation from the 20VC interview as well. My small reflection: The words “coding” and “programming” are about to evolve. The way we write software will keep shifting — but the need for people who understand how to build software won’t go away. If anything, demand goes up. Whether you write code by hand, guide AI to write it, or combine both — the real value is in thinking, specifying clearly, and pushing innovation forward. "Even earlier this year, we saw some senior business leaders advise people to not learn to code on the grounds that AI will automate it. We'll look back on that as some of the worst career advice ever given. As coding becomes easier with AI assisting us, a lot more people should learn to code, not fewer. I mentioned the mockery example just now with building it up for feedback swiping, but I think for a lot of job functions, people that know how to tell a computer exactly what they want it to do so the computer can do it for you, they'll just be more powerful. For the foreseeable future, the language of precisely telling computers what you want it to do is coding. It doesn't mean you should write code by hand. Writing code by hand is becoming obsolete. Really, don't do that. But get AI to write code for you. If people could do that, it would be more effective and more powerful and have more fun." - Andrew Ng

8

I came across a new benchmark technique CAR-bench(29 Jan 2026) for evaluating LLM agents — and it highlights why production reliability is still hard Over the past year, LLM-based agents have made impressive progress. Tool use, reasoning traces, and multi-step planning all look promising in demos. I recently came across CAR-bench, a new benchmark technique designed to evaluate LLM agents the way we actually deploy them. Unlike traditional benchmarks that assume: • Clear user intent • Perfect tool availability • One-shot success CAR-bench introduces realism: • Ambiguous and underspecified user requests • Unsatisfiable tasks (missing tools or missing data) • Strict domain and safety policies • Multi-turn, stateful interactions • Repeated executions to measure consistency The benchmark evaluates three task types: • Base tasks – solvable, end-to-end workflows • Hallucination tasks – where the correct behavior is to explicitly refuse • Disambiguation tasks – where agents must pause and clarify before acting This leads to a critical evaluation shift: • Pass@k → measures capability • Pass³ → measures reliability And the gap between them is large. Even frontier “thinking” models often: • Act prematurely without resolving ambiguity • Violate policies inconsistently across runs • Fabricate or silently mask missing information • Succeed once, but fail repeatedly One of the most important findings is that the dominant failure mode is not lack of reasoning — it is decision timing. A key conclusion from the paper is that for production systems, the model alone is not the most important factor — a stable, well-designed system architecture with explicit control and safety layers is equally critical. CAR-bench also surfaces a real cost vs reliability trade-off: • Reasoning models improve results • But significantly increase token usage, latency, and inference cost • And even with more reasoning, disambiguation remains largely unsolved The benchmark strongly reinforces a production lesson many teams learn the hard way: Reliable agentic systems require architecture, not just better models. In practice, this means: • Separating intent understanding, planning, and execution • Using deterministic policy and safety layers • Treating ambiguity and missing information as blocking states • Measuring consistency, not best-case accuracy CAR-bench is not without limitations: • Users are simulated, not real humans • The domain is automotive (though failure patterns generalize) • Interaction is text-only (real voice systems are harder) • Tool coverage, while realistic, is not exhaustive This post is a high-level summary. If you want to read the full paper, find attachment with the post. Next: I’ll share how these CAR-bench ideas can be applied to current enterprise AI and agentic systems—including evaluation, guardrails, and multi-agent design. Stay tuned. #AgenticAI #GenAI #LLMBenchmarks #AIArchitecture #MLOps #ResponsibleAI #ProductionAI

17

Just shipped a new version of the Phase Bridge Faster, leaner, and ready for real-world use: • 10–50× faster I/O with binary formats (MessagePack, CBOR, NPZ) • ~50% less memory thanks to float32 + lazy loading (no upfront allocation) The core is now more efficient and stable—paving the way for applied use cases. Next up: expanding modality support to graphs, vectors, embeddings, and beyond. Fully backward-compatible, docs updated, and security unchanged (hashes + end-to-end verification still in place). Smooth upgrade—go ahead and update! https://lnkd.in/dkbsgiBB

2

When an LLM underperforms, the instinct is often to fine-tune it. But fine-tuning is not where most performance problems originate. In real systems, LLM failures usually come from data design, not model capacity. What that looks like in practice: - Multiple tasks collapsed into a single instruction dataset - Inconsistent labels or supervision across sources - No explicit failure taxonomy to distinguish reasoning, retrieval, or formatting errors - Evaluations that hide where the model actually breaks Fine-tuning doesn’t resolve these issues. It amplifies the existing signal in the data good or bad. If the signal is noisy or misaligned, the fine-tuned model becomes confidently wrong. Recent research consistently shows that LLM performance gains are driven more by data quality, structure, and curation than by the fine-tuning method itself, poorly curated or misaligned datasets can negate improvements, degrade general capabilities, and introduce safety regressions, while cleaner, task-aligned data often outperforms additional fine-tuning. (CLEAR framework; recent EMNLP & ACL findings on data-centric AI and fine-tuning trade-offs) What this means for teams building production systems: - Fine-tuning should not be the first response to quality issues - Data pipelines and evals usually determine the ceiling before tuning does - Many model problems disappear once task boundaries and supervision are fixed Fine-tuning is valuable but only after: - The task is stable - Failure modes are well understood - Data quality and coverage are already strong Fine-tuning is a scaling lever, not a debugging tool. so the Right question must be What evidence do you rely on before deciding that fine-tuning is truly necessary? #LLM #AIEngineering #DataCentricAI #FineTuning #MLOps #MachineLearning #AgenticSystems

6

Temperature is the most misunderstood LLM parameter. I see engineers set it to 0 for 'accuracy.' I see others crank it to 1 for 'creativity.' The truth: Temperature 0 = deterministic but brittle Temperature 0.7 = balanced for most tasks Temperature 1+ = chaotic for structured outputs For extraction tasks: 0-0.2 For generation tasks: 0.5-0.8 For brainstorming: 0.8-1.0 Context determines temperature. Not preference. #corporatebhaiya #resume #interview #mentorship #job #internship #workshop #JobAlert #HiringNow #IndiaJobs #CareerOpportunity #JobVacancy #JobOpening #JobSearchIndia #Hiring #CareersInTech #connections #JoinOurTeam #interested #career #roadmap

4