Best Practices For Data Backup And Recovery

Your data has a temperature, and you are wasting money if you don't know it. Hot, Warm, and Cold data. Storing data is not just about saving it and forgetting about it. You need to understand how often you will access the data and how long you should keep it. You can group data into three categories based on how often it's accessed: 𝗛𝗼𝘁 𝗗𝗮𝘁𝗮 • What It Is: Data that you need often and fast. • Where It's Stored: On fast storage like SSDs or even in memory. • Examples: Things like product recommendations or cached search results. • Cost: Storing hot data is expensive, but accessing it is cheap because it's always ready to go. 𝗪𝗮𝗿𝗺 𝗗𝗮𝘁𝗮 • What It Is: Data you access occasionally, like once a month. • Where It's Stored: On slower but still accessible storage, e.g., Amazon S3 Infrequently Accessed Tier, Google Nearline. • Examples: Older logs or data that are not as frequently needed. This could be data that you use for reporting or analytics. • Cost: It is cheaper to store than hot data, but accessing it costs a bit more. 𝗖𝗼𝗹𝗱 𝗗𝗮𝘁𝗮 • What It Is: Data are rarely accessed and primarily kept for long-term storage. • Where It's Stored: On the cheapest storage options, like HDDs or cloud archive services. • Examples: Old backups or records that you keep for compliance reasons. • Cost: It is very cheap to store but can be slow and expensive to access. Retention is a different animal, explains "how long you should keep data" and is based on 4 pillars: 𝗩𝗮𝗹𝘂𝗲 Is this data critical for you, or can it be recreated if needed? You should keep Important data for longer. 𝗧𝗶𝗺𝗲 For data you store in fast-access places like memory, set a time limit (TTL) for how long it stays there before moving it to cheaper storage. 𝗖𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲 Some laws require you to keep data for a certain amount of time or delete it after a specific period. Make sure your data storage practices follow these rules. 𝗖𝗼𝘀𝘁 Storing data costs money. To save on storage costs, you can automate deleting or archiving data when it's no longer needed. Don't just store data—manage it. Save this for your next Storage Decision.

This might sound odd coming from me but data is not always a valuable asset. While it might be a liability worth holding onto against future opportunities for some time, it’s a mistake to think that data has inherent value and is hence something to be hoarded across the board. Why a liability? Because at a minimum any data you hold needs to be stored, to be backed up, to have auditable access control and to be kept safe from cyber attacks. Watching from the periphery of data-intensive industries over the past decade, it would have been easy to have heard the message ‘storage is cheap now’ but to have missed the later trend of ‘so we will generate and store WAY WAY more data than we did before, to the point that the bills mount up again’. So how do you figure out whether data ‘X’ is valuable enough to retain? First you need an estimation of the cost to retain. I would start by drafting a data retention framework that covers at a minimum 🔍 categorisation of data into sensitivity levels with enough articulation of those levels so folk can map both data that exists and data that is touted for collection to the appropriate level 🔍 retention durations for each of those sensitivity levels (that are compliant with regulation if regulated and compliant with clearly and publicly articulated periods if not) 🔍 storage options for both operational and analytically focussed data with links to current costing for each option 🔍and of course, a timetable and process for updating said framework!!!! That gives you something to baseline at least a guesstimate of the raw storage costs of holding on to whatever data you might be considering stockpiling. Not a total cost of ownership cost of course but multipliers for personnel and overhead costs can help out with that at a guesstimate level. Now the liability label is more self evident as that data is something you need to pay regular amounts of money to hang on to on an ongoing basis. How substantial are the upside benefits of the data you are thinking about storing, be that through lower support costs, cross sales, targeted marketing, lower churn via stickier products, data sharing partnerships or actual new revenue streams via value added and hence higher priced products? Sure it’s all guesstimates and whiteboard estimates but it’s better than storing everything, sleeping through your next 36 months of inexorably increasing cloud storage bills and making the front page of the newspaper for all the wrong reasons when the hackers look your way. All before the value you were contemplating makes its way off the backlog. Photo by Fabian Blank on Unsplash

After more of a decade of cleaning up Azure enviroments! here are some of my tips (outside of RI and rightsize SKUs which everyone else talks about is the common tips..) ✅ Check for Orphaned Resources (Disks, App Service Plans, SQL, PIP) Use the Orphaned Resources Workbook in Azure Monitor. ) Do some manual checks to see which workloads are actually being used. ✅ Workloads that are stopped but not deallocated? That means that hardware is still reserved and the cost is still running. Either deallocate or delete them. ✅ Check for Old snapshots of disks (So many old test and lab enviroments here) ✅ Check for backup tier (Do you need GRS?) or is ZRS, which is 40% cheaper good enough? ✅ If you do not need lower then 24 hours RTO on Backup, use standard policy and not enhanced. (While enhanced is the only one supported for Premium v2 disks but is a lot more expensive) ✅ What kind of redundancy do you need on your storage? LRS/ZRS/GRS? Adding GRS provides higher redundancy but higher cost and latency. Use LRS on active storage and use higher-level on backup data. ✅ Check for disk type (Many disks can be configured with SSDv2 tier which can be cheaper and faster in many cases) ✅ Check for which logs are actually needed? A AKS Cluster alone can generate close to 23 GB a month without little workloads. Unfortunately few have a good strategy around logs. What is needed and why? Just by disabling kube-admin can save you much. This applies for all services and workloads you have. ✅ Check for which logs are needed for security? By enabling Sentinel on a Log Analytics workspace you are paying for the data ingested and not how many analytics rules you have. So much logs are just being collected without being used. ✅ Do you need all Defender for Cloud SKUs enabled? Defender for Storage has a cost of 10$ per account. Make sure you use Defender for Cloud service where it matters! production workloads) ✅ Can you use Workspaces with API Management instead of having multiple production API Management instances? ✅ DDoS on IP instead of Network. In most cases you require DDoS protection on certain external services but not everything. Using IP based protection is a lot cheaper compared to Network based protection. ✅ What kind of storage do you need? Azure has different NFS/SMB based storage options providing much of the same capabilities such as Azure Files and Azure NetApp files, but there is a high cost difference between them. ✅ Do you need Private Endpoints? (Cost per PE, Cost bandwidth cost, Cost VNET Peering) or can services (storage) be locked down using Service Endpoint with or without policies? ✅ Is LicenseType = "Windows_Client" set on AVD machines? It does it automatically using the portal, but not via Terraform/Bicep. This ensures that you do not pay for Windows license for AVD workloads. ✅ Standalone or centralized services? I see so often redundant instances APIM, WAF, NAT, Backup vaults, try and avoid redundant services

I co-authored a patent on backup in the cold storage at Druva, so when Toshal Khawale asked me about AWS Backup cold tier costs, I genuinely thought I already knew the answer. Cold storage is one of those things that feels completely obvious at face value - data you rarely touch moves to a cheaper tier, you pay less per GB, your bill drops and everyone's happy. That's the whole idea behind cold storage and it works exactly that way on pretty much every platform I've seen. Except AWS Backup does something I didn't expect and honestly hadn't seen before, and it completely breaks that logic for EBS backups specifically. When EBS snapshots transition to cold storage in AWS Backup, each incremental snapshot gets silently converted into a full snapshot. So if you have a 1 TB volume and you're only changing 50 GB a month, warm storage keeps those incrementals small, you're storing 1,550 GB total across 12 monthly backups at steady state. But the moment that same backup plan transitions to cold tier, AWS stores 12 full copies of the entire volume, which means 12,000 GB instead of 1,550 GB, and the 75% per-GB discount you were expecting gets completely wiped out by the nearly 8x increase in stored data volume. I ran the actual numbers on a 1 TB EBS volume with a 5% monthly change rate and the annual cost comes out to $930 in warm tier versus $1,830 in cold tier - so the tier that's supposed to be cheaper ends up costing you almost double, and the kicker is that AWS Cost Explorer won't flag this, your backup plan won't warn you, and the bill just quietly climbs. The break-even change rate where cold tier actually starts making sense is 18.2% monthly block change and most production EBS workloads, databases, application servers, file systems, sit somewhere between 2-10%, which means cold tier is actively hurting the majority of teams who've enabled it thinking they were saving money. I wrote up the full analysis with the break-even math, the services where cold storage actually does help (EFS and DynamoDB are genuinely different stories), and six alternatives that actually reduce the bill without the penalty. Link in the first comment. Has anyone else hit something like this where enabling what looks like a cost saving feature quietly made things worse? CloudYali #FinOps #AWS #CloudCost #DevOps #AWSBackup

How I Reduced Our GCP Development Database Cost from $300 → $15 While reviewing our GCP bills recently, I noticed something odd — our development environment was consuming over $300/month just for Cloud SQL (PostgreSQL). After digging in, I found that: • GCP’s Enterprise edition quietly enforces a minimum 60 GB storage limit. • High availability (HA) was turned on by default, doubling compute cost. • Point-in-time recovery, daily automated backups, and replica settings were enabled even though they weren’t needed for dev. For any non-production environment, these features add zero real value but multiply your bill. If you use Cloud SQL for development: • Set availability type to Zonal (Single zone) • Disable point-in-time recovery • Keep backups manual or short retention • Reduce to the smallest tier (db-g1-small or custom minimal vCPU/RAM) Even after tuning, Cloud SQL Enterprise still wouldn’t allow storage below 60 GB — so I took another route. I spun up a small e2-micro VM on Compute Engine, installed PostgreSQL 15, restricted access to specific IPs, and automated daily dumps with pg_dump. Now, the same dev database runs perfectly fine at $10–15/month — a 94% cost reduction with full control. If you’re running light workloads or dev/staging databases, don’t overlook this. Sometimes “managed” isn’t the most efficient — especially when you just need something lean and simple. #optimization #cloudsql #gcp #developer

I reduced my client's S3 bill by 36% in 10 minutes. Here's how you can too. The Situation: My client had significant S3 costs. After a quick audit, I discovered they were using Standard storage for everything—even data that was rarely (or never) accessed. Their use cases were simple: ❗️Audit logs stored for compliance (never retrieved) ❗️Backup streams for their data lake (only for reconciliation) ❗️Archives they'd forgotten about The Solution: I built two Python scripts to audit their buckets, analyze sizing, and review lifecycle policies. Then I implemented intelligent tiering based on actual access patterns: ✔️ S3 Intelligent Tiering for infrequent access (55% savings) ✔️ S3 Glacier Flexible Retrieval with 90-day minimum (82% savings) ✔️ S3 Glacier Deep Archive with 180-day minimum (96% savings) ✔️ 30-day expiration policy for all lower environments The Result: 36% cost reduction with zero impact on operations. Total time invested: 10 minutes of automation + implementation. The Lesson: Most companies are overpaying for S3 because they default to Standard storage. A simple audit and lifecycle policy can unlock massive savings without impacting operations. Stop treating all your data the same. Not everything needs instant access. #AWS #FinOps #DevOps #TechLeadership #success

“Freed 78% storage overnight. Saved $50k.” Sounds heroic. Until you realize… you deleted 4 years of transaction history before an audit. This is the difference between cost optimization and data governance. Archiving ≠ Deleting. S3 backup ≠ Compliance strategy. Script ≠ System design. In real systems: Financial data has retention policies (often 5–7 years). Auditable systems need immutability and traceability. Deletes require legal + accounting alignment. Archival needs versioning, lifecycle rules, and restore validation. A Python script can free storage. But it can also free you from your job. The real engineering move isn’t “delete older than 90 days.” It’s: → Define retention policy → Partition large tables → Use cold storage tiers properly → Implement soft deletes → Validate restore paths → Align with compliance teams Optimization without context is just risk. In production, data is liability, asset, and legal evidence all at once. Before you celebrate storage saving: Ask: “Can I defend this in an audit?” #SoftwareEngineering #DataEngineering #Postgres #SystemDesign #TechLeadership

🚨 “𝐇𝐨𝐰 𝐚 𝐒𝐢𝐦𝐩𝐥𝐞 𝐉𝐞𝐧𝐤𝐢𝐧𝐬 𝐉𝐨𝐛 𝐂𝐨𝐬𝐭 𝐔𝐬 𝐓𝐡𝐨𝐮𝐬𝐚𝐧𝐝𝐬 𝐨𝐟 $$$$—𝐀𝐧𝐝 𝐇𝐨𝐰 𝐈 𝐒𝐭𝐨𝐩𝐩𝐞𝐝 𝐈𝐭!” 🚨⁣ ⁣ Last month, my manager was shocked when we received our AWS invoice. The bill was sky-high—thanks to 85,000 GB of old snapshots piling up over the last year, costing us Thousands of $Dollars annually. 😱⁣ Here’s what happened:⁣ - My team used a Jenkins job to take snapshots of EC2 instances before making changes (like patching or upgrading RHEL).⁣ - These snapshots were tagged with `adhoc=True` for identification, but there was no automation to clean them up.⁣ - Over time, thousands of snapshots piled up, and no one noticed… until the bill arrived. 💸⁣ ⁣ What Did I Do?⁣ I rolled up my sleeves and created a Python script using Boto3 to:⁣ 1. Scan for snapshots older than 7 days with the `adhoc=True` tag.⁣ 2. Delete them and store their details in a CSV file.⁣ 3. Send an email notification to the team with the cleanup report.⁣ Github Link: https://lnkd.in/eaJyd6tt ⁣ Then, I integrated this script into a Jenkins pipeline and scheduled it to run every night at 10 PM⁣ ⁣ The Result?⁣ ✅ All piled-up snapshots were deleted.⁣ ✅ Thousands of $Dollars in annual AWS storage costs saved.⁣ ✅ A happy manager and a relieved team. 😊⁣ ⁣ Key Takeaways⁣ 1. Automation is Key: Manual processes often lead to overlooked costs. Automate wherever possible.⁣ 2. Monitor Your Cloud Resources: Regularly audit your cloud environment to avoid unexpected bills.⁣ 3. Tagging is Powerful: Use tags effectively to identify and manage resources.⁣ ⁣ 💬 Your Turn: ⁣ Have you ever faced a similar situation where automation saved the day? Or ⁣do you have tips for optimizing cloud costs? Share your stories in the ⁣ comments!⁣ ⁣ 🔗 **Save this post** for later and tag someone who’s dealing with cloud cost optimization!⁣ ⁣ #CloudComputing #AWS #CostOptimization #DevOps #Automation #Python #Boto3 #Jenkins #TechStories

"Isn’t the cloud supposed to be cheaper?" That’s the question I hear all the time—usually right after someone gets their AWS bill and starts questioning their life choices. A while back, someone asked me the same thing. I smiled, knowing where this was going. They had just received their bill, and let’s just say—it wasn’t giving “cost savings.” More like, “who spent my salary before I even touched it?” Cloud costs can spiral out of control not because AWS is expensive, but because someone, somewhere, forgot to turn things off. Or worse—resources were deployed with the mindset of, “We’ll optimize later.” Spoiler alert: Later never comes. As an AWS Solutions Architect, a big part of my job is helping businesses design architectures that scale efficiently—without setting their budgets on fire. Here’s how I approach it: 1️⃣ Right-Sizing: Not Every Workload Needs a Mansion I’ve seen companies run massive EC2 instances for tiny applications. Imagine renting a five-bedroom duplex just to store your suitcase—that’s how some workloads treat AWS compute. ✅ Fix: Always match compute resources to actual demand. Use EC2 Auto Scaling, AWS Compute Optimizer, and Savings Plans to avoid over-provisioning. 2️⃣ Storage Sprawl – The "Just Keep It" Syndrome S3 is cheap, but keeping every single log, backup, and memes from 2016 adds up. Some teams treat storage like a black hole—once data enters, it never leaves. ✅ Fix: Use S3 Lifecycle Policies to automatically archive or delete old data. Leverage Glacier for long-term storage at a fraction of the cost. 3️⃣ Unused Resources – The Silent Bill Killers Sometimes, an EC2 instance is launched for a quick test and then… forgotten. It sits there, silently racking up costs like a gym membership you swore you’d cancel. ✅ Fix: Set up AWS Budgets and Cost Anomaly Detection to catch unused resources. Implement scheduled shutdowns for non-production environments. 4️⃣ Data Transfer Costs – The "Surprise" on Your Bill Cross-region data transfers can be sneaky. A team once ran an application where data constantly moved between regions—each transfer was tiny, but at scale? The bill told a different story. ✅ Fix: Optimize network architecture using VPC endpoints, CloudFront caching, and regional service placements to minimize data transfer fees. The Cloud Isn’t Expensive—Bad Architecture Is AWS offers the tools to optimize costs—you just have to design with cost efficiency in mind from day one. A well-architected cloud environment doesn’t just scale—it scales smartly. What’s the biggest AWS billing shock you’ve ever seen? Let’s discuss. #AWS #CloudComputing #AWSBilling #CostOptimization #FinOps #CloudArchitecture #AWSCommunity #Presales #CloudEngineering

𝗟𝗮𝘀𝘁 𝗺𝗼𝗻𝘁𝗵, 𝗜 𝘄𝗮𝘁𝗰𝗵𝗲𝗱 𝗮 𝘀𝘁𝗮𝗿𝘁𝘂𝗽 𝗯𝘂𝗿𝗻 𝘁𝗵𝗿𝗼𝘂𝗴𝗵 $𝟱𝟬,𝟬𝟬𝟬 𝗶𝗻 𝘀𝘁𝗼𝗿𝗮𝗴𝗲 𝗰𝗼𝘀𝘁𝘀 𝗯𝗲𝗰𝗮𝘂𝘀𝗲 𝘁𝗵𝗲𝘆 𝗽𝘂𝘁 𝗘𝗩𝗘𝗥𝗬𝗧𝗛𝗜𝗡𝗚 𝗶𝗻 𝗵𝗼𝘁 𝘀𝘁𝗼𝗿𝗮𝗴𝗲. Their 7-year-old log files were getting the same premium treatment as their real-time user data. Here's what I learned helping them fix their storage architecture: 🔥 𝗛𝗼𝘁 𝘃𝘀 𝗖𝗼𝗹𝗱 𝗶𝘀𝗻'𝘁 𝗮𝗯𝗼𝘂𝘁 𝘁𝗲𝗺𝗽𝗲𝗿𝗮𝘁𝘂𝗿𝗲, 𝗶𝘁'𝘀 𝗮𝗯𝗼𝘂𝘁 𝗮𝗰𝗰𝗲𝘀𝘀 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀 Most engineers think "fast = good, slow = bad." Wrong mindset. Hot storage costs $120K/year for 100TB. Cold storage? $1,200. Same data, 99% cost reduction. The key is understanding when you actually need microsecond access versus when minutes or hours is perfectly fine. 💡 𝗬𝗼𝘂𝗿 𝗱𝗮𝘁𝗮 𝗵𝗮𝘀 𝗮 𝗹𝗶𝗳𝗲𝗰𝘆𝗰𝗹𝗲, 𝘁𝗿𝗲𝗮𝘁 𝗶𝘁 𝗹𝗶𝗸𝗲 𝗼𝗻𝗲 Stop treating all data equally. Set up automated policies from day one. Recent user activity data stays hot for 30 days, moves to warm storage for 90 days, then archives to cold storage. Let the system handle the transitions automatically instead of manual management nightmares. 📊 𝗠𝗼𝗻𝗶𝘁𝗼𝗿 𝗮𝗰𝗰𝗲𝘀𝘀 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀, 𝗻𝗼𝘁 𝗴𝘂𝘁 𝗳𝗲𝗲𝗹𝗶𝗻𝗴𝘀 I see teams making storage decisions based on "what if we need this data quickly someday." Start logging every data access from the beginning. You'll discover that 80% of your "critical" data hasn't been touched in months. Real usage patterns will shock you. 🏗️ 𝗛𝘆𝗯𝗿𝗶𝗱 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲𝘀 𝘄𝗶𝗻 𝗲𝘃𝗲𝗿𝘆 𝘁𝗶𝗺𝗲 Netflix doesn't keep every movie in hot storage. Popular content stays hot, everything else moves to cheaper tiers. Your architecture should work the same way. Build intelligent caching layers that promote frequently accessed data and demote the rest. ⚡ 𝗛𝗶𝗱𝗱𝗲𝗻 𝗰𝗼𝘀𝘁𝘀 𝘄𝗶𝗹𝗹 𝗱𝗲𝘀𝘁𝗿𝗼𝘆 𝘆𝗼𝘂𝗿 𝗯𝘂𝗱𝗴𝗲𝘁 Storage price per GB is just the beginning. Retrieval fees, data transfer costs, minimum billing periods, and operational overhead add up fast. Calculate total cost of ownership, not just the sticker price. The startup I mentioned? They implemented intelligent tiering and cut their storage costs by 85% while actually improving performance for their users. What's been your biggest storage architecture lesson? Have you seen teams make similar mistakes? #DataArchitecture #CloudStorage #Engineering #TechStrategy