Future Trends in Recommendation Algorithms

Your team built the wrong recommender. Not wrong because of the model. Wrong because of the assumption baked into the architecture. At Twitter, we hit a hard ceiling on the scaling timeline ranking. The culprit: Two-Tower retrieval treating user history like an unordered grocery list. But human behavior isn't a set of independent clicks. It's a chronological sentence. And order is the meaning. Classic Retrieve-then-Rank breaks down since it's context-blind. It mathematically treats: watching a setup tutorial after buying a tool, the same as watching it before buying. This deletes the "grammar" of user intent. To tackle this, the paradigm has been shifting from "Retrieve and Rank" to "Encode and Generate". Think of it like framing recommendations as an LLM predicting the next token. → The Encoder: A Transformer compresses chronological interaction history into a dense context vector - not a demographic profile, but a real-time "state of mind." → The Decoder: An autoregressive decoder generates recommendations sequentially. Each suggestion is conditioned on both user history and what was just recommended. This is how Pinterest's PinRec captures long-range item dependencies that point-wise ranking entirely misses. The same principles apply to SOTA models from Google and Meta. Here's the execution reality though: this paradigm shifts the bottleneck from your Vector DB to your GPU, so you're trading retrieval speed for temporal accuracy. You also have a hard constraint that you cannot autoregressively generate thousands of items in real-time. But the field has already engineered around this. Read how Semantic IDs help in part 1: https://lnkd.in/g75sgtxd To summarize: the field is converging on three trends - unifying retrieval and ranking under shared generative architectures, integrating preference-aligned reward-driven learning, and rapidly adopting multimodal foundation models. This is Part 4 of my Generative RecSys series. Next up: I'll break down how to overcome context window limitations in Generative RecSys design. The era of pure Search-and-Rank is over. If you've experimented with sequential or generative recommenders, what was your biggest surprise?

A sneak peek into my notes from the "LLMs for Recommendation Systems" private event hosted by Meta 📝 Engineers and researchers are all converging on the same architectural bet: move parts of retrieval/ranking toward LLM-native sequence modeling! → History repeats itself. In 2016, "deep learning won't work in recsys" was the consensus. Today, LLMs get the same skepticism. We are at the cusp of a new baseline. → Semantic IDs are the new token. Represent catalog items as hierarchical discrete tokens - often derived via residual/hierarchical quantization methods such as RQ-VAE or RQ-KMeans - that LLMs can natively generate and rank over. This is a primitive that everything else builds on. → Spotify's GLIDE injects long-term user embeddings as soft prompts, letting the model balance stable preferences with shifting intent at inference time, without expanding the text prompt or retraining the base model for each request. LinkedIn's work similarly points toward LLM/SLM-based ranking, distillation, context compression, and hybrid text-embedding interactions. → YouTube's PLUM adds continued pre-training on domain-specific interaction data before generative retrieval fine-tuning. Key finding: LLM initialization + CPT beats random init, general sequence modeling transfers to recommendation. → Meta's HSTU treats user interaction history as a token sequence with attention and relative positional bias, scaling with LLM-like power laws. Generative retrieval predicts the next item's Semantic ID autoregressively - blurring or partially unifying retrieval and ranking. Meta is very bullish on this. → Netflix's direction is especially interesting: build a member-understanding foundation model, then post-train/adapt per application. The separation of “understand the member” from “solve the task” feels like the right abstraction. And my favorite practical pattern of the day: use a cheap CPU-based MLP to pre-filter features before handing anything to the LLM. Counter-features still matter. Role-based summarization of user activity for prompt compression is a real technique people are shipping, not a research idea. Premature optimization is still the root of all evil. But the teams not experimenting with LLM-native retrieval and ranking right now are going to have some catching up to do. #RecSys #LLMRecSys

Exciting Innovation in LLM-Based Recommendations! I just read a fascinating paper titled "Rethinking LLM-Based Recommendations: A Query Generation-Based, Training-Free Approach" from researchers at KAIST. This work addresses critical challenges in using Large Language Models for recommendation systems. Current LLM-based recommendation methods face several limitations: - Inefficiency with large candidate pools - Sensitivity to item positioning in prompts (the "lost in the middle" phenomenon) - Poor scalability - Unrealistic evaluation methods using random negative sampling The researchers propose an innovative solution called Query-to-Recommendation (QUEREC), which takes a fundamentally different approach: >> How QUEREC Works Instead of the traditional method of feeding candidate items into prompts for reranking, QUEREC leverages LLMs to generate personalized queries that directly retrieve relevant items from the entire candidate pool. This eliminates the need for candidate pre-selection entirely! The framework operates through several key components: 1. Item Query Generation: The LLM analyzes item metadata and user reviews to generate queries that capture the distinctive features of each item. 2. User Query Generation: The system creates personalized queries based on user history and preferences. 3. Similarity-based Retrieval: Using a pre-trained text encoder, the system computes similarity scores between user and item representations. 4. Divergent Perspective Reranking: QUEREC combines insights from both LLM-generated queries and traditional collaborative filtering models to produce the final recommendations. >> Technical Advantages What makes this approach particularly impressive: - Training-Free Implementation: QUEREC can be integrated into existing ID-based recommendation systems without additional training. - Parallel Architecture: Unlike traditional serialized pipelines where LLMs rerank pre-selected candidates, QUEREC operates in parallel with traditional recommendation models, allowing both to extract top-k items independently from the entire item pool. - Enhanced Diversity: Experiments showed QUEREC produces more balanced distribution of recommended items compared to conventional models that exhibit bias toward specific item groups. - Improved Performance for Minor Items: The approach significantly outperforms existing methods for items that appear less frequently in training sets. This approach represents a significant advancement in recommendation systems, offering a more efficient, scalable, and diverse approach to personalized recommendations. The training-free nature makes it particularly valuable for rapidly evolving recommendation environments.

For years, we've forced knowledge graphs into recommender systems, hoping their structure would magically yield explanations. Usually, it doesn't. We get accuracy gains, but the "why" remains trapped in vector space - a statistical ghost, not a logical chain. A new research work titled "Evolutionary Reinforcement Learning for Explainable Recommendation on Knowledge Graph", aces at it !!! Here’s what I found most compelling: 1) The "Mutation" hack: To navigate huge decision spaces, the AI doesn't just pick the top-ranked options. It intentionally mutates its list - swapping a few obvious choices for high-potential "dark horses." It's a brilliant, biologically-inspired trick to avoid local optima and stay creative. 2) The stunning (and puzzling) result: On most datasets, it beats state-of-the-art models by ~2-3%. But on the sparse, messy Amazon Cell Phones dataset, performance exploded: +51% Precision, +44% Hit Rate. This suggests the model isn't just a lab benchmark winner - it might be a secret weapon for noisy, real-world data where obvious patterns fail. 3) The honest limitation: The entire elegant system depends on a clean, structured Knowledge Graph (the map of connections between users, items, and features). The authors openly admit that building and maintaining this "map" is the hard, expensive, human part. The AI is a brilliant navigator, but it needs a good map. 4) The future vision: They propose teaming this system with Large Language Models. Let the RL agent find the rigorous, causal path. Then let the LLM translate that path into fluent, human-friendly language. This splits the work perfectly: reliability for the machine, articulation for the machine. This isn't just another accuracy bump. It's a philosophical shift - treating the "why" as a first-class output, not a post-hoc justification. The pressing question it leaves us with: If explainability at this level requires pristine knowledge graphs, how do we build and maintain them at scale in our messy, ever-changing digital world? The algorithm is ready. Is our data infrastructure? #ExplainableAI #XAI #ReinforcementLearning #KnowledgeGraph #RecommenderSystems #MachineLearning #AIResearch #DataScience #TechEthics

🎯 The New Battleground for OTTs: AI-Led Content Discovery is the Differentiator By Vishal Arya | Architecting the Future of AI & Entertainment In a world where content is abundant, but attention is limited, the greatest challenge for Over-the-Top (OTT) platforms is not streaming; it’s ensuring the right stories are surfaced at just the right moment. Whether managing 10,000 titles or a million, the harsh reality remains: your best content remains unseen until it is discovered. Welcome to the age where discovery isn’t a UX feature—it’s an AI product. Here's how next-gen tech is rewriting the playbook: 🔍 1. AI-Generated Metadata: The New Fuel for Discovery Engines Forget static tags. Today’s LLMs extract sentiment, tone, narrative arcs, and character dynamics—transforming raw content into rich, machine-readable signals. 💡 Real-World Impact: A short-video platform used GenAI to auto-suggest titles and summaries. When creators adopted these, CTRs jumped 7.1%, while average watch time rose 4.1%. Metadata isn’t just a label—it’s a conversion driver. 🤖 2. Multimodal Recommendation Systems: Beyond Clicks & Views Modern recommendation engines blend text + vision + audio embeddings to capture a user’s content preferences more holistically. 🎥 Think: Transformers that understand mood, tone, setting—not just genre or actor. 🔐 3. Cross-Platform Behavioral Modelling: Breaking the App Silo In super-aggregated OTT ecosystems, federated learning is the secret sauce. It enables shared personalisation across apps—without sharing user data. 🎞 4. AI-Driven Media Optimization: From Upload to Upsell Predictive AI now scores content for genre affinity, retention risk, watchability, and trend fit. 🧠 Platforms are using this to auto-select thumbnails, assign content badges (“must-watch,” “comfort content”), and even sequence UI placement dynamically. 🔥 Result: One global streamer saw 35% higher engagement and 22% better retention with predictive content scoring + automated UI asset testing. 🕹 5. Gamified & Mood-Based Discovery: Swipes. Quizzes. Emotions. Next-gen OTT UX is borrowing from gaming and social. AI-powered interfaces respond to real-time behavior with interactive cards, quizzes, mood filters, and emotion-based content sorting. 🎮 Edutainment Win: Platforms with gamified discovery saw 18% longer sessions, better content depth exploration, and higher rewatch ratios. 🧠 Final Word from the C-Suite: The content itself isn’t king anymore. Discovery is. In an AI-first world, attention is earned by platforms that understand behavior, context, and emotion in real time. At the heart of the next OTT revolution is a new stack: agentic AI, dynamic metadata, real-time UX, and semantic intelligence. If you're still relying on legacy recommender engines, you’re already behind. The winners are turning their discovery engines into intelligent, evolving ecosystems.

For years, recommendation systems have largely followed the same blueprint: retrieve, rank, rerank, recommend. The models have become more sophisticated, but the overall pipeline has remained surprisingly stable. This survey on Agentic Recommender Systems argues we're entering a new phase where recommendations are no longer produced by a single ranking model, but by AI agents that can reason, plan, use tools, maintain memory, and even collaborate with other agents. Rather than viewing LLMs as just another ranking component, the paper frames them as decision-makers with varying levels of autonomy – from augmenting existing recommenders to eventually orchestrating the entire recommendation process. What I found most interesting wasn't just the agentic architecture. It was also the discussion around evaluation. Many recent papers introduce increasingly capable agents, yet continue evaluating them with traditional ranking metrics like NDCG or Recall@k. Those metrics tell us whether the final recommendations were good, but they reveal very little about how the agent reached those recommendations. If an agent searches external knowledge, calls tools, revises its plan, or learns from previous interactions, the reasoning process becomes part of the system being evaluated, not just the final ranked list. That suggests recommendation evaluation may need to evolve from measuring only outcomes to also measuring decision trajectories: – Did the agent retrieve trustworthy information? – Did it use the right tools? – Was its planning efficient? – Did memory improve future recommendations? – What was the latency and cost of reasoning? We've already seen a similar evolution in agent benchmarks, where execution traces are becoming as important as final accuracy. It seems recommendation systems are beginning to follow the same path. As recommendation engines become agentic, we're no longer evaluating just a ranking function. We're evaluating a reasoning system. Paper: https://lnkd.in/ezaZU6CH #AI #LLMs #RecommendationSystems #AgenticAI #InformationRetrieval #MachineLearning

Netflix just revealed they're applying Large Language Model principles to recommendation systems at scale. Their foundation model processes hundreds of billions of user interactions - comparable to the token volume of ChatGPT and other LLMs. What's fascinating is how they're "tokenizing" your viewing history. Just as LLMs convert text into tokens, Netflix transforms your binge sessions into meaningful sequences that capture your preferences. But unlike language models where each token has equal weight, Netflix weights a 2-hour movie watch differently than a 5-minute trailer browse. The technical innovation comes in addressing the "cold start" problem - recommending new shows before anyone's watched them. They've developed a hybrid approach that blends metadata-based embeddings with learnable ID embeddings through an attention mechanism based on content "age." New titles rely more on metadata until enough user interaction data accumulates. Their confirmation that the same scaling laws governing LLMs apply to recommendation systems too is interesting. Their performance graphs show consistent improvements as model size increases, mirroring what we've seen with language models. Will foundation models eventually replace all specialized ML systems across industries? Could the next breakthrough in recommendation come from merging content understanding with user behavior prediction? Full article link in comments. #AIforRecommendation #FoundationModels #MachineLearning #NetflixTech

Most recommendation systems today are a patchwork of specialized models — one for search, one for collaborative filtering, one for user understanding. Spotify's NEO paper at KDD 2026 asks a genuinely uncomfortable question: what if that architectural fragmentation is the problem, not the solution? The paper introduces NEO, a decoder-only LLM adapted into a tool-free, catalog-grounded generator that handles recommendation, search, and user understanding within a single sequence model. The key mechanism is language-steerability: text prompts control the task, the target entity type, and the output format — IDs, free text, or mixed — while constrained decoding guarantees that generated items are always valid catalog entries. Items are represented as Semantic IDs (SIDs), treated as a distinct modality and integrated via staged alignment and instruction tuning. The system was evaluated on a real-world catalog of over 10 million items across multiple media types. Why does this matter to a CTO or Head of Research at a MarTech company? Three reasons. First, the orchestration tax of multi-model pipelines is real — every handoff between a retrieval model, a ranker, and a user understanding module introduces latency, error propagation, and optimization barriers. NEO demonstrates that a single model can achieve cross-task transfer, meaning that training on recommendation improves search performance and vice versa. Second, language-steerability is not a UX feature — it is an architectural primitive. The ability to condition a generative model on natural language instructions at inference time is precisely what makes AI systems adaptable to the kind of long-tail, context-specific targeting that platforms like SOMIN have been pursuing through Omni-Sourced User Profiling. Third, the SID-as-modality framing is a meaningful contribution to the broader question of how to ground LLMs in structured, domain-specific entity spaces without sacrificing generative flexibility — a problem directly relevant to anyone building on top of large catalogs. From my own research trajectory — starting with multi-source user profile learning at ICMR 2015, through cross-domain recommendation via multi-layer graph clustering at SIGIR 2017, and into the explainability work we published at ACM MM 2023 and 2025 — the direction NEO takes feels like a natural convergence point. The challenge was always that user signals are fragmented across modalities and tasks. What NEO proposes is not just a model architecture but a reframing: treat the user's intent as a language-expressible instruction, and let a single grounded generator handle the rest. The question of whether this holds at the scale and heterogeneity of real marketing catalogs — where items span creative assets, audience segments, and campaign objectives — remains open. Key takeaways: - A single language-steerable generative model can outperform task-specific baselines on recommendation, search, and user understanding simultane

In the next decade, products that adapt to individuals will win on retention, loyalty, and long-term engagement. This is why I am betting on the impact LLMs will have in RecSys. If you want to understand where this is heading, these are a few papers since 2022 that genuinely moved the needle. GPT4Rec: A Generative Framework for Personalized Recommendation Instead of directly scoring items, the model generates natural-language representations of user intent and uses those to retrieve relevant items. It’s one of the clearest examples of how LLMs enable semantic, interpretable personalization rather than opaque scoring functions. https://lnkd.in/e6tF5ee2 TALLRec: An Effective Tuning Framework to Align LLMs with Recommendation Tasks TALLRec shows that general-purpose LLMs don’t automatically make good recommenders. What matters is alignment. With lightweight, task-specific tuning, LLMs can meaningfully outperform zero-shot approaches, making them viable components in real recommender pipelines. https://lnkd.in/e3GjJaDs GLoSS: Generative Language Models with Semantic Search for Sequential Recommendation This work combines LLMs with semantic retrieval to improve sequential recommendation, especially in cold-start and sparse-data settings. The key insight is that semantic understanding of items and histories often beats strict ID-based matching. https://lnkd.in/eCKZ49Cx Lost in Sequence: Do LLMs Understand Sequential Recommendation? A reality check paper. It shows that naïvely feeding user histories into LLMs doesn’t mean the model actually understands temporal preference shifts. The paper introduces mechanisms to inject sequential structure explicitly, highlighting that personalization is as much about time as it is about content. https://lnkd.in/eAGBPHWD Text Is All You Need: Learning Language Representations for Sequential Recommendation This paper helps bridge traditional recommender systems and language models by treating user–item interactions as text sequences. It strongly influenced later work by showing that recommendation can be framed as a language modeling problem without abandoning rigor. https://lnkd.in/eEYRy_7S When you step back, a few clear trends emerge from these ideas. First, personalization is becoming semantic. User preferences are no longer just vectors, they’re expressed and reasoned about in language. This opens the door to explainable and interactive recommenders. Second, retrieval plus generation is the dominant pattern. LLMs don’t replace recommendation pipelines, they enhance them by generating intents, enriching retrieval, and improving ranking decisions. Third, sequence awareness is non-negotiable. Understanding how preferences evolve over time is critical, and LLMs need explicit structure to do this well. Finally, alignment beats scale. Bigger models alone don’t solve personalization. The real gains come from aligning LLMs with recommendation objectives and user behavior.

Bringing together complex systems, network science, and computational social science yields powerful, vital insights on human-AI coevolution. This important paper by leaders including Albert-Laszlo Barabasi and Alex 'Sandy' Pentland describes the vital role of feedback in human-AI systems and the implications. Some of the key findings: 🔁 Feedback loops are the engine of coevolution. Human-AI coevolution is driven by a self-reinforcing feedback loop: users generate data that train AI, which in turn influences users’ future choices. This dynamic can amplify unintended consequences, such as polarization, bias, or system degradation. The loop is especially evident in recommender systems and generative AI models. 🌍 Society-centered AI reframes the debate. The authors introduce “society-centred AI” as a new paradigm, expanding beyond technology- or human-centred approaches. It emphasizes that the feedback loop's impact spans individual to societal levels, and managing it requires scientific innovation, legal regulation, and political action, not just more technology. 📉 Generative AI risks collapsing into sameness. In content generation, when LLMs are fine-tuned on AI-generated data, they regress to the mean—standardizing language and reducing diversity. This “autophagy” loop risks a slow erosion of linguistic and cultural variation. 📈 Recommenders reinforce inequality and monopolies. "Recommenders" - systems in retail, media, or navigation that guide users - increase efficiency for individuals but can reduce diversity and favor dominant players at scale. For instance, they may concentrate traffic on certain routes or drive sales toward popular items—reinforcing market concentration and inequality. 🏙️ AI can backfire on cities. Navigation apps like Google Maps aim to reduce individual travel time, but coevolution causes collective issues. Too many users being routed through “optimal” paths creates new congestion, longer travel times, and higher emissions. ⚖️ Ownership of recommenders shapes power. The authors highlight a modern twist on Marx’s theory: owning the “means of recommendation” (e.g., recommender platforms) confers immense influence. In the absence of governance, this could entrench digital monopolies, skew public discourse, and diminish collective welfare. 🚫 Techno-solutionism is insufficient. The paper cautions against the belief that more or better AI will fix AI’s harms. Without societal oversight and a shift from individual utility to collective benefit, the feedback loop will likely continue to deepen existing social divides, especially along lines of race, income, and geography.