LlamaIndex

How many people show up on a Tuesday night to talk about document processing for AI agents? 🤔 In New York, enough to fill the room and start a waitlist. In San Francisco, almost 600. We were in both cities on the same night to talk about one problem: agents don't read documents the way people do. Look at an invoice and you can tell the monthly rate from the total in a second. An agent often gets the words without the layout, so it has to guess. In New York, we partnered with KERNEL on Braintrust's Agent Builders Night. In San Francisco, we partnered with WorkOS on Daytona's AI Builders. Thanks to them for having us, and to Abrar Mahi and Yong Park from our team for speaking!

We're #hiring a new Developer Evangelist, Technical Content in San Francisco, California. Apply today or share this post with your network.

Yesterday I hosted a fun dinner conversation with Vincent Sunn Chen from Snorkel AI on evals and RL environments. The "data and RL env" companies (like Snorkel) have seen massive growth in the past few years. There's been an explosion of interest in evals. At the same time, models are ripping through benchmarks with each new release. We talked about the evals everyone is defining, what evals are still left unsolved, what’s left up to frontier models vs. intelligence that you own, and more: - A big challenge for building RL environments is “fairness” - when the model fails on a given environment, can you attribute it to the input, harness, or reward model? - Building proper rewards is hard. Some tasks are not easily quantifiable. You also want to discourage reward hacking. At the same time, you don’t want to be too prescriptive with intermediate rewards. - Long horizon evals are still extremely hard, some business processes can take up to weeks or months before the final outcome - Most regulated industries still need human in the loop to guarantee ~100% accuracy, “80%” accuracy is not good enough - As models get more intelligent, there will be a barbell of boutique data vendors (e.g. any SMB) any scaled up data providers. - Models still exhibit “jagged intelligence” where they still fail on a long tail of edge cases. - There might always be opportunities to gather unique data for a given task and posttrain models for lower cost and higher accuracy. This marks #003 in our founder dinner series. What topic should we discuss next? Let us know your thoughts below! Preston Yadegar Faraz Siddiqi Akshaya Jagadeesh Ali Ebrahim Shahul Elavakkattil Shereef Robert Stewart Jithin James Abhijeet Shenoi Vincent Sunn Chen

We're #hiring a new Enterprise Account Executive - EMEA in London, England. Apply today or share this post with your network.

Document parsing requires a lot of on-the-fly decision making. Can Jev make those decisions for you? How closely can OSS alternatives land? We explored the early approaches to Jev and Jev-like models on several documents tasks like orientation detection, language detection, routing, and more. https://lnkd.in/gHuWievF And of course, LlamaParse already handles these same tasks for you 🦙

Parsing forms still trips up the latest frontier VLMs. A form isn't text on a page. It's a set of fields, grouped into sections, each tied to a specific box. That's why forms need purpose-built parsing, not a bigger general model: ✅️ Detect every field, not just the obvious ones ✅️ Keep the hierarchy of sections and fields ✅️ Tie every value to the exact box it came from ✅️ Understand handwriting and checkmarks Our latest blogpost breaks down where VLMs fail on real W-2s, 1040s, W-9s and scanned W-4s, and a custom cookbook for LlamaParse to handle them at a fraction of the cost 👇 https://lnkd.in/gzBPmtBa

We're #hiring a new Developer Evangelist, Growth in San Francisco, California. Apply today or share this post with your network.

We're #hiring a new Member of Technical Staff, Backend in San Francisco, California. Apply today or share this post with your network.