API Integration Challenges

A candidate interviewing for a Senior Engineer @ Meta was asked to design a rate limiter. Another candidate at Google's L5 loop got hit with the same question. I've been asked this three times across different companies. Rate-limiting questions look simple until you add one layer of complexity: – Add distributed rate limiting? Now you're dealing with race conditions and clock skew. – Add multiple rate limit tiers? Welcome to priority queues and quota management. – Add per-user, per-IP, and per-API-key limits? Your Redis bill just exploded. Here's my personal checklist of 15 things you must get right when building rate limiters: 1. Always do rate limiting on the server, not the client → Client-side limits are useless. They’re easily bypassed, so always enforce limits on your backend. 2. Choose the right placement → For most web APIs, place the rate limiter at the API gateway or load balancer (the “edge”) for global protection and minimal added latency. 3. Identify users correctly → Use a combination of user ID, API key, and IP address. Apply stricter limits for anonymous/IP-only clients, higher for authenticated or premium users. 4. Support multiple rule types → Allow per-user, per-IP, and per-endpoint limits. Make rules configurable, not hardcoded. 5. Pick an algorithm that fits your needs → Know the pros/cons: –  Fixed Window: Easy, but suffers from burst issues. –  Sliding Log: Accurate, but memory-heavy. –  Sliding Window Counter: Good balance, small memory footprint. – Token Bucket: Handles bursts and steady rates, an industry standard for distributed systems. 6. Store rate limit state in a fast, shared store → Use an in-memory cache like Redis or Memcached. Every gateway instance must read and write to this store, so limits are enforced globally. 7. Make every check atomic → Use atomic operations (e.g., Redis Lua scripts or MULTI/EXEC) to avoid race conditions and double-accepting requests. 8. Shard your cache for scale → Don’t rely on a single Redis instance. Use Redis Cluster or consistent hashing to scale horizontally and handle millions of users/requests. 9. Build in replication and failover → Each cache node should have replicas. If a primary fails, replicas take over. This keeps the system available and fault-tolerant. 10. Decide your “failure mode” → Fail-open (let all requests through if the cache is down) = risk of backend overload. Fail-closed (block all requests) = user-facing downtime. For critical APIs, prefer fail-closed to protect backend. 11. Return proper status codes and headers → Use HTTP 429 for “Too Many Requests.” Include headers like: – X-RateLimit-Limit,  – X-RateLimit-Remaining,  – X-RateLimit-Reset, Retry-After This helps clients know when to back off. 12. Use connection pooling for cache access → Avoid reconnecting to Redis on every check.  Pool connections to minimize latency. Continued in Comments...

Few Lessons from Deploying and Using LLMs in Production Deploying LLMs can feel like hiring a hyperactive genius intern—they dazzle users while potentially draining your API budget. Here are some insights I’ve gathered: 1. “Cheap” is a Lie You Tell Yourself: Cloud costs per call may seem low, but the overall expense of an LLM-based system can skyrocket. Fixes: - Cache repetitive queries: Users ask the same thing at least 100x/day - Gatekeep: Use cheap classifiers (BERT) to filter “easy” requests. Let LLMs handle only the complex 10% and your current systems handle the remaining 90%. - Quantize your models: Shrink LLMs to run on cheaper hardware without massive accuracy drops - Asynchronously build your caches — Pre-generate common responses before they’re requested or gracefully fail the first time a query comes and cache for the next time. 2. Guard Against Model Hallucinations: Sometimes, models express answers with such confidence that distinguishing fact from fiction becomes challenging, even for human reviewers. Fixes: - Use RAG - Just a fancy way of saying to provide your model the knowledge it requires in the prompt itself by querying some database based on semantic matches with the query. - Guardrails: Validate outputs using regex or cross-encoders to establish a clear decision boundary between the query and the LLM’s response. 3. The best LLM is often a discriminative model: You don’t always need a full LLM. Consider knowledge distillation: use a large LLM to label your data and then train a smaller, discriminative model that performs similarly at a much lower cost. 4. It's not about the model, it is about the data on which it is trained: A smaller LLM might struggle with specialized domain data—that’s normal. Fine-tune your model on your specific data set by starting with parameter-efficient methods (like LoRA or Adapters) and using synthetic data generation to bootstrap training. 5. Prompts are the new Features: Prompts are the new features in your system. Version them, run A/B tests, and continuously refine using online experiments. Consider bandit algorithms to automatically promote the best-performing variants. What do you think? Have I missed anything? I’d love to hear your “I survived LLM prod” stories in the comments!

New Update: Amazon DSP campaign and creative APIs are now generally available. This is a build on many of the announcements from #unBoxed2024 What is it? This new feature allows users to create, read, and update their Amazon DSP campaigns, ad groups, targets, and creatives through a programmatic interface. How does it work? These APIs enable technology providers and advertisers to develop custom experiences within their own applications and seamlessly run Amazon DSP campaigns within existing workflows. The new APIs can be used in conjunction with existing audience and deal resources, providing a comprehensive toolkit for end-to-end campaign management. Users can now store Amazon DSP campaign data locally, simplify campaign and creative creation, and automate optimizations to maximize campaign performance. Why should I care? This update is a game-changer for Amazon DSP users. Here's why it matters: 1. Efficiency boost: Streamline your campaign and creative creation process, significantly reducing activation time for new campaigns. 2. Better data control: Store and manage Amazon DSP campaign data locally, giving you more control over your data and analytics. 3. Custom optimization: Automate optimizations across campaign, ad group, and targeting settings, allowing for data-driven decisions on bids and budgets. 4. Seamless integration: Easily integrate Amazon DSP into your existing tech stack, enabling you to track campaigns in your own tools and sync campaign metadata with your data storage solutions. 5. Performance improvement: Experiment with new audiences and quickly remove underperforming ones to maximize campaign performance. 6. Real-time adjustments: Automatically adjust bids and budgets in real-time, ensuring your campaigns are always performing at their best. Bottom line: Whether you're a large agency or tech partner looking to integrate Amazon DSP more deeply into your operations or an individual advertiser seeking to automate and optimize your campaigns, these new APIs offer exciting possibilities to enhance your advertising efforts on Amazon's platform. Want to check it out? You can learn more about these new features at the Amazon Ads website (https://lnkd.in/gESdMWhy). For those ready to dive in, check out the developer guide (https://lnkd.in/gQAPRdcs) and reference documentation (https://lnkd.in/gBV-BbVb) to start leveraging these powerful new APIs in your advertising strategy.

Security can’t be an afterthought - it must be built into the fabric of a product at every stage: design, development, deployment, and operation. I came across an interesting read in The Information on the risks from enterprise AI adoption. How do we do this at Glean? Our platform combines native security features with open data governance - providing up-to-date insights on data activity, identity, and permissions, making external security tools even more effective. Some other key steps and considerations: • Adopt modern security principles: Embrace zero trust models, apply the principle of least privilege, and shift-left by integrating security early. • Access controls: Implement strict authentication and adjust permissions dynamically to ensure users see only what they’re authorized to access. • Logging and audit trails: Maintain detailed, application-specific logs for user activity and security events to ensure compliance and visibility. • Customizable controls: Provide admins with tools to exclude specific data, documents, or sources from exposure to AI systems and other services. Security shouldn’t be a patchwork of bolted-on solutions. It needs to be embedded into every layer of a product, ensuring organizations remain compliant, resilient, and equipped to navigate evolving threats and regulatory demands.

Vulnerabilities in MCP (Model Context Protocol) I was hired to audit integrations of an LLM with MCP, for use with data management tools, log collections and automated routines. Here are some problems I found and would like to share so that those of you who want to implement MCP in your products can start thinking about security at the beginning of the development cycle. However, it is worth mentioning that there are still not many efficient solutions, despite some selling LLM Firewalls. I would like to test and validate the effectiveness of this. Anyway, let's get to the points: 1) The lack of HTTPS in API Integrations was a problem I noticed a lot. The LLM and the integrated MCP APIs that were integrated with the tools or executed commands and received the response to the commands allowed me to view the requests and responses. I used Wireshark to validate. 2) Inadequate Permission Management, allowing me to access data from other clients without any tenant isolation, all via Prompt Injection and Burp Suite to analyze requests and perform basic manipulations. 3) Abuse of Automations and Unrestricted Resource Consumption, allowing me to trigger multiple parallel routines, all via a single prompt, or sending different prompts causing the server to trigger routines all at once, without proper thread queue management. I used Burp Suite with Intruder and created a list of prompts and executed at least 50 different prompts with the same context. In addition, there was no control over the request limit in the APIs. 4) SQL Injection via Prompt, basically making requests using human language, for example: “what columns does the users table have?” resulted in queries being executed directly without control and spitting out information, i.e., it seems that the integration opened the database schema (weird). Obviously, the problem is that it built the query in the backend and processed it as an SQL query. I used Burp Suite in this case to analyze the response, etc. 5) Hardcoded Secrets in the MCP Code. API tokens, database credentials, and endpoints were found directly in the MCP integration scripts. Although it is obvious, just because they are in the backend does not mean they must be hardcoded. Unfortunately, I was unable to extract secrets via prompt injection or obtain an RCE. 6) Broad Context allowing Full Control of the application. Although I did not obtain the application secrets, providing broad context to the LLM gave it full control over the integrated systems, executing tasks that should be exclusive to the admin, since the configured keys had excessive permissions that allowed the execution of numerous functions. In short, these are flaws that a trained developer with knowledge of application security could resolve, but many who start integrating solutions with AI do not worry about Shift-Left. #mcp #AI #redteam #cybersecurity #AISecurity #mcpsecurity #pentest #llmpentest

As SaaS vendors scale, integration requirements shift from “nice to have” to mission-critical. But in parallel, the demands of enterprise IT - data residency, compliance, performance, and cost predictability, only become more stringent. At Integration App, we’re addressing this tension head-on by delivering a universal integration layer that runs directly within your infrastructure. Unlike hosted integration solutions or embedded iPaaS platforms that introduce new data flows, latency layers, and vendor-side operational dependencies, our model prioritizes infrastructure sovereignty. You retain full control over how and where integrations execute while benefiting from a platform that automates and abstracts the complexity of connecting to thousands of third-party systems. Here's what that unlocks: 1. Data Sovereignty by Default No proxies. No data egress. Customer data never leaves your environment. Whether you’re in a private VPC, on-prem, or operating under industry-specific compliance regimes (HIPAA, SOC 2, GDPR, FedRAMP), our deployment model ensures your security posture isn’t compromised by integration complexity. 2. Security and Compliance-First Architecture Deploy integrations in line with your own IAM policies, access control frameworks, and encryption standards. All executions occur in your trusted compute environment, enabling full auditability and adherence to internal and external governance requirements. 3. Infrastructure-Native Deployment The integration layer is designed to be deployed alongside your core application stack, whether containerized via Kubernetes or integrated into a custom CI/CD pipeline. 4. Performance Without Penalties Since integration flows run at the edge of your application stack, you avoid the latency and variability introduced by centralized middleware or external orchestration layers. 5. Predictable, Scalable Economics No usage-based throttling. No per-flow billing. With a flat pricing model and no API call metering, you can scale integration volume without introducing infrastructure cost uncertainty. This predictability becomes critical as integration use cases grow across customers, tenants, and third-party systems. AI-Augmented, API-Agnostic By decoupling Integration App logic from specific APIs, and using AI to generate contextual, app-specific execution paths, we eliminate the bottlenecks of manual, one-off integrations.

🛑 "429 Too Many Requests" isn't just an error code; it's a survival strategy for your distributed systems. Stop treating Rate Limiting as a simple counter. To prevent crashes, you need the right algorithm. This visual explains the patterns you need to know. 𝐇𝐨𝐰 𝐰𝐞 𝐜𝐨𝐮𝐧𝐭: 1️⃣ Token Bucket: User gets a "bucket" of tokens that refills at a constant rate. Great for bursty traffic. If a user has been idle, they accumulate tokens and can make a sudden burst of requests without being throttled immediately. Use Case: Social media feeds or messaging apps. 2️⃣ Leaky Bucket: Requests enter a queue and are processed at a constant, fixed rate. Acts as a traffic shaper. It smooths out spikes, protecting your database from write-heavy shockwaves. Use Case: Throttling network packets or writing to legacy systems. 3️⃣ Fixed Window: A simple counter resets at specific time boundaries (e.g., the top of the minute). Easiest to implement but suffers from the "boundary double-hit" issue (e.g., 100 requests at 12:00:59 and 100 more at 12:01:01). Use Case: Basic internal tools where precision isn't critical. 4️⃣ Sliding Window Log: Tracks the timestamp of every request. Solves the boundary issue completely. It’s highly accurate but expensive on memory (O(N) space complexity) because you store logs, not just a count. Use Case: High-precision, low-volume APIs. 5️⃣ Sliding Window Counter: The hybrid approach. Approximates the rate by weighing the count of the previous window and the current window. Low memory footprint, high accuracy. Use Case: Large-scale systems handling millions of RPS. 𝐖𝐡𝐞𝐫𝐞 𝐰𝐞 𝐞𝐧𝐟𝐨𝐫𝐜𝐞 6️⃣ Distributed Rate Limiting: Essential for microservices. You cannot rely on local memory; you need a centralized store (like Redis with Lua scripts) to maintain a global count across the cluster. 7️⃣ Fixed Window with Quota: Often distinct from technical throttling. This is business logic—hard caps over long periods (months/years). Use Case: Tiered billing plans (e.g., "Free Tier: 10k calls/month"). 8️⃣ Adaptive Rate Limiting: The "smart" limiter. It doesn't use static numbers but monitors system health (CPU, memory, latency). If the system struggles, it tightens the limits automatically. Use Case: Auto-scaling systems and disaster recovery. 𝐖𝐡𝐨 𝐰𝐞 𝐥𝐢𝐦𝐢𝐭 9️⃣ IP-Based Rate Limiting: The first line of defense. Limits based on the source IP to prevent botnets or DDoS attacks. Use Case: Public-facing unauthenticated APIs. 🔟 User/Tenant-Based Rate Limiting: Limits based on API Key or User ID. Ensures one heavy user doesn't degrade performance for others ("Noisy Neighbor" problem). Use Case: SaaS platforms and multi-tenant architectures. 💡 For most production systems, Sliding Window Counter combined with Distributed Limiting is the gold standard. It offers the best balance of memory efficiency and user fairness. #SystemDesign #SoftwareArchitecture #API #Microservices #DevOps #BackendEngineering #RateLimiting #CloudComputing

Literally one of the best ways you can build LLM-based applications: Mirascope is an open-source library. Big selling point: This is not going to force abstractions down your throat. Instead, Mirascope gives you composable primitives for building with large language models. For example, • You can incorporate streaming into your application • Add support for tool calling • Handle structure outputs You can pick and choose what you need without worrying about unnecessary abstractions. Basically, this is a well-designed low-level API that you can use like Lego blocks. For example, attached, you can see a streaming agent with a tool calling in 11 lines of code. There are no magic classes here, or hidden state machines that do things you don't know about. This is just simple Python code. A few highlights: • It supports OpenAI, Anthropic, Google, and any other model. • You can swap providers by changing a string. • Agents are just tool calling in a while loop. • You always decide how your code operates. • Type-safe end-to-end. • Great autocomplete, catches errors before runtime. Mirascope is fully open source, MIT-licensed. Here is the GitHub repository: https://lnkd.in/eKeuqHww

The challenge of integrating multiple large language models (LLMs) in enterprise AI isn’t just about picking the best model, it’s about choosing the right mix for each specific scenario. When I was tasked with leveraging Azure AI Foundry alongside Microsoft 365 Copilot, Copilot Studio, Claude Sonnet 4, and Opus 4.1 to enhance workflows, the advice I heard was to double down on a single, well‑tuned model for simplicity. In our environment, that approach started to break down at scale. Model pluralism turned out to be the unexpected solution, using multiple LLMs in parallel, each optimised for different tasks. The complexity was daunting at first, from integration overhead to security and governance concerns. But this approach let us tighten data grounding and security in ways a single model couldn’t. For example, routing the most sensitive tasks to Opus 4.1 helped us measurably reduce security exposure in our internal monitoring, while Claude Sonnet 4 noticeably improved the speed and quality of customer‑facing interactions. In practice, the chain looked like this: we integrated multiple LLMs, mapped each one to the tasks it handled best, and saw faster execution on specialised workloads, fewer security and compliance issues, and a clear uplift in overall workflow effectiveness. Just as importantly, the architecture became more robust, if one model degraded or failed, the others could pick up the slack, which matters in a high‑stakes enterprise environment. The lesson? The “obvious” choice, standardising on a single model for simplicity, can overlook critical realities like security, governance, and scalability. Model pluralism gave us the flexibility and resilience we needed once we moved beyond small pilots into real enterprise scale. For those leading enterprise AI initiatives, how are you balancing the trade‑off between operational simplicity and a pluralistic, multi‑model architecture? What does your current model mix look like?

You're in a backend interview. They ask: "Design a rate limiting system for an API used by millions. Where do you start?" Here’s how you impress 👇 1.Start with the goal: Prevent abuse, ensure fair usage, and protect system stability. 2. Choose a rate-limiting strategy:   - Fixed window   - Sliding window   - Token bucket (most flexible for millions of users)   - Leaky bucket 3. Decide on the granularity:   - Per user? Per IP? Per API key?   - Define the limit (e.g. 100 req/min) 4. Pick a fast, distributed store:   Use Redis to track usage—fast reads/writes, with key expiry support. 5. Implementation flow:   - Each request checks Redis.   - If within limit → continue.   - Else → return 429 Too Many Requests. 6. Make it scalable:   - Use Redis clusters for horizontal scaling.   - Hash keys for sharding.   - Push logic to API gateways like Kong or Cloudflare Workers for global edge enforcement. 7. Add burst control and global override flags:   Just in case you need to protect key endpoints during load spikes.