Kubernetes Cluster Management

3 weeks back while I was working on a Kubernetes project, saw a very common bad practice, i.e. image size is 8 GB which could have been just 436 MB (Multi-stage Build)!!!! After working on such numerous projects, here I have made a list of best practices for containers: Use Minimal Base Images: Start with minimal base images like Alpine Linux or Distroless to reduce the attack surface and improve container startup times. Single Concern per Container: Each container should have only one responsibility. If an application has multiple components (e.g., web server, database etc), they should be split into separate containers and managed as separate services. Stateless Applications: Design your applications to be stateless as much as possible. This allows Kubernetes to easily scale, restart, or replace containers without losing data. Liveness and Readiness Probes: Use liveness probes to let Kubernetes know when to restart a container and readiness probes to know when a container is ready to start accepting traffic. Resource Limits: Set resource requests and limits for CPU and memory to ensure that the container gets its required resources and doesn't consume more than it should. Security: Run containers with a non-root user. Use network policies to control communication between pods. Regularly scan container images for vulnerabilities. Use Kubernetes RBAC (Role-Based Access Control) to limit permissions. Immutable Containers: Avoid making changes to running containers. Instead, create a new container image and deploy it. This ensures consistency across environments. Use Labels and Annotations: Use labels for organizing and selecting groups of resources. Annotations can be used to store additional metadata. Configurations and Secrets: Use ConfigMaps for non-sensitive configuration data and Secrets for sensitive data. Avoid hardcoding configurations in the container image. Logging and Monitoring: Ensure that your applications log to the standard output and standard error streams. This allows Kubernetes to handle and redirect the logs appropriately. Integrate with monitoring tools like Prometheus to keep an eye on the health and performance of your containers. Regularly Update and Patch: Regularly update your container images to include security patches and updates. Use image scanning tools to identify and fix vulnerabilities. Graceful Shutdown: Ensure that your applications handle the SIGTERM signal and shut down gracefully. This allows them to finish processing current requests and release resources before shutting down. Avoid Using latest Tag: Be explicit with container image tags. Avoid using the latest tag as it can lead to unpredictable deployments. Storage Considerations: If your application needs persistent storage, use Persistent Volumes (PV) and Persistent Volume Claims (PVC) in Kubernetes. Ensure that the storage solution you choose is compatible with the dynamic nature of containerized deployments. Follow Sandip Das for more!

Using unverified container images, over-permissioning service accounts, postponing network policy implementation, skipping regular image scans and running everything on default namespaces…. What do all these have in common ? Bad cybersecurity practices! It’s best to always do this instead; 1. Only use verified images, and scan them for vulnerabilities before deploying them in a Kubernetes cluster. 2. Assign the least amount of privilege required. Use tools like Open Policy Agent (OPA) and Kubernetes' native RBAC policies to define and enforce strict access controls. Avoid using the cluster-admin role unless absolutely necessary. 3. Network Policies should be implemented from the start to limit which pods can communicate with one another. This can prevent unauthorized access and reduce the impact of a potential breach. 4. Automate regular image scanning using tools integrated into the CI/CD pipeline to ensure that images are always up-to-date and free of known vulnerabilities before being deployed. 5. Always organize workloads into namespaces based on their function, environment (e.g., dev, staging, production), or team ownership. This helps in managing resources, applying security policies, and isolating workloads effectively. PS: If necessary, you can ask me in the comment section specific questions on why these bad practices are a problem. #cybersecurity #informationsecurity #softwareengineering

99% of teams are overengineering their Kubernetes deployments. They choose the wrong tool and pay for it later lol After managing 100+ Kubernetes clusters and debugging 100s of broken deployments, I’ve seen most teams picking up Helm, Kustomize, or Operators based on popularity, not use case. (1) 𝗜𝗳 𝘆𝗼𝘂’𝗿𝗲 𝗱𝗲𝗽𝗹𝗼𝘆𝗶𝗻𝗴 <10 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀 → 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝗛𝗲𝗹𝗺 ► Use public charts only for commodities: NGINX, Cert-Manager, Ingress. ► Always fork & freeze charts you rely on. ► Don’t template environment-specific secrets in Helm values. Cost trap: Over-provisioned replicas from Helm defaults = 25–40% hidden spend. Always audit values.yaml. (2) 𝗪𝗵𝗲𝗻 𝘆𝗼𝘂 𝗵𝗶𝘁 𝗺𝘂𝗹𝘁𝗶𝗽𝗹𝗲 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀 → 𝗦𝘄𝗶𝘁𝗰𝗵 𝘁𝗼 𝗞𝘂𝘀𝘁𝗼𝗺𝗶𝘇𝗲 ► Helm breaks when you need deep overlays (staging, perf, prod, blue/green.) ► Kustomize is declarative, GitOps-friendly, and patch-first. ► Use base + overlay patterns to avoid value sprawl. ► If you’re not diffing kustomize build outputs in CI before every push, you will ship misconfigs. Pro tip: Pair Kustomize with ArgoCD for instant visual diffs → you’ll catch 80% of config drift before prod sees it. (3) 𝗦𝘁𝗮𝘁𝗲𝗳𝘂𝗹 𝘄𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀 & 𝗱𝗼𝗺𝗮𝗶𝗻 𝗹𝗼𝗴𝗶𝗰 → 𝗢𝗽𝗲𝗿𝗮𝘁𝗼𝗿𝘀 𝗼𝗿 𝗯𝘂𝘀𝘁 ► Operators shine when apps manage themselves: DB failovers, cluster autoscaling, sharded messaging queues. ► If your app isn’t managing state reconciliation, an Operator is expensive theatre. But when you need one: Write controllers, don’t hack CRDs. Most “custom” Operators fail because the reconciliation loop isn’t designed for retries at scale. Always isolate Operator RBAC (they’re the #1 privilege escalation vector in clusters.) 𝐌𝐲 𝐇𝐲𝐛𝐫𝐢𝐝 𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤 At 50+ services across 3 regions, we use: ► Helm → Install “standard” infra packages fast. ► Kustomize → Layer custom patches per env, tracked in GitOps. ► Operators → Manage stateful apps (DBs, queues, AI pipelines) automatically. Which strategy are you using right now? Helm-first, Kustomize-heavy, or Operator-led?

One of the biggest hurdles to mastering Kubernetes isn't just the complexity, it’s the fear of a massive cloud bill at the end of the month. Many beginners stick to local tools like Minikube, but there is no substitute for the experience of working with managed services like Google Kubernetes Engine (GKE). In this latest tutorial, I break down exactly how to spin up a fully functional GKE cluster on Google Cloud for less than the price of a couple of coffees per month. ☕️ Why this approach is a game-changer for your DevOps journey: - GKE Autopilot: Pay only for the pods you run, not for idle infrastructure. - Terraform-Powered: Learn Infrastructure as Code (IaC) by deploying and destroying clusters with a single command. - Security First: Includes best practices like Workload Identity and auto-upgrades right out of the box. - Cost-Saving Hacks What you’ll walk away with: ✅ A repeatable, production-ready Kubernetes setup. ✅ A GitHub starter repo to kickstart your own projects. ✅ The confidence to experiment in a real cloud environment without breaking the bank. If you’re a new GCP user, you can even use your $300 free credits to run this setup entirely for free for 90 days. Watch the full tutorial here: https://lnkd.in/gW4Ec8dN Let’s stop making excuses and start building!

I’ve spent 7 years obsessing over the perfect Kubernetes Stack. These are the best-practices I would recommend as a basis for every Kubernetes cluster. 1. Implement an Observability stack A monitoring stack prevents downtime and helps with troubleshooting. Best-practices: - Implement a Centralised logging solution like Loki. Logs will otherwise disappear, and it makes it easier to troubleshoot. - Use a central monitoring stack with pre-built dashboards, metrics and alerts. - For microservices architectures, implement tracing (e.g. Grafana Tempo). This gives better visibility in your traffic flows. 2. Setup a good Network foundation Networking in Kubernetes is abstracted away, so developers don't need to worry about it. Best practices: - Implement Cilium + Hubble for increased security, performance and observability - Setup a centralised Ingress Controller (like Nginx Ingress). This takes care of all incoming HTTP traffic in the cluster. - Auto-encrypt all traffic on the network-layer using cert-manager. 3. Secure your clusters Kubernetes is not secure by default. Securing your production cluster is one of the most important things for production. Best practices: - Regularly patch your Nodes, but also your containers. This mitigates most vulnerabilities - Scan for vulnerabilities in your cluster. Send alerts when critical vulnerabilities are introduced. - Implement a good secret management solution in your cluster like External Secrets. 4. Use a GitOps Deployment Strategy All Desired State should be in Git. This is the best way to deploy to Kubernetes. ArgoCD is truly open-source and has a fantastic UI. Best practices: - Implement the app-of-apps pattern. This simplifies the creation of new apps in ArgoCD. - Use ArgoCD Autosync. Don’t rely on sync buttons. This makes GIT your single-source-of-truth. 5. Data Try to use managed (cloud) databases if possible. This makes data management a lot easier. If you want to run databases on Kubernetes, make sure you know what you are doing! Best practices - Use databases that are scalable and can handle sudden redeployments - Setup a backup, restore and disaster-recovery strategy. And regularly test it! - Actively monitor your databases and persistent volumes - Use Kubernetes Operators as much as possible for management of these databases Are you implementing Kubernetes, or do you think your architecture needs improvement? Send me a message, I'd love to help you out! #kubernetes #devops #cloud

🚀 Install and Bootstrap a High-Availability Kubernetes Cluster with Ansible As a Platform Engineer, I wanted a fast, reproducible way to deploy a fully functional HA Kubernetes cluster across multiple VMs. No more manual setup or inconsistent environments—just one command to get everything running. Using Ansible, I automated: 🔹 kubeadm init with controlled configuration (CIDRs, load balancer) 🔹 Automatic master & worker node joins 🔹 Core components: Cilium (CNI), Ingress-NGINX, Cert-Manager, OpenEBS 🔹 Local kubeconfig ready for kubectl and CI/CD pipelines Why this matters: ✅ Fast, consistent HA clusters for dev & QA ✅ Standardized networking, ingress, certificates, and storage ✅ Idempotent and re-runnable—safe to redeploy anytime ✅ Simplified onboarding: one doc + one command This setup drastically reduces lead time for new environments while keeping everything predictable and versioned. 📂 Check out the playbooks and instructions on GitHub: https://lnkd.in/eWHpVrBb 📽️ Watch Demo: https://lnkd.in/egFj_iuA 👇 If this is useful, hit ‘Like’ and share your thoughts or questions!” #CKA #CKS #CKAD #Kubestraunaute #Kubernetes #Ansible #HACluster #Cilium #IngressNGINX #CertManager #OpenEBS #DevOps #PlatformEngineering

🚀 𝗘𝗞𝗦 𝗖𝗹𝘂𝘀𝘁𝗲𝗿 𝗕𝘂𝗶𝗹𝗱 𝗨𝘀𝗶𝗻𝗴 𝗧𝗲𝗿𝗿𝗮𝗳𝗼𝗿𝗺 𝗼𝗻 𝗔𝗪𝗦 – 𝗦𝘁𝗲𝗽-𝗯𝘆-𝗦𝘁𝗲𝗽 𝗚𝘂𝗶𝗱𝗲 ☁️⚙️ This workflow covers everything from setting up the AWS provider to deploying a sample application on Kubernetes. 🔹 𝗞𝗲𝘆 𝗦𝘁𝗲𝗽𝘀 𝗖𝗼𝘃𝗲𝗿𝗲𝗱: ✅ Create Terraform project structure ✅ Configure AWS Provider ✅ Build VPC with public & private subnets ✅ Create IAM Roles & Policies ✅ Provision EKS Cluster ✅ Configure Managed Node Groups ✅ Initialize & Validate Terraform Code ✅ Deploy Infrastructure with Terraform ✅ Configure kubectl Access ✅ Verify Cluster Health ✅ Deploy Sample Nginx Application ✅ Clean Up Resources 💡 𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗕𝗲𝘀𝘁 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀 • Store Terraform state remotely (S3 + DynamoDB) • Use private subnets for worker nodes • Enable IRSA (IAM Roles for Service Accounts) • Maintain separate workspaces for Dev/UAT/Prod • Always run terraform fmt, validate, and plan before apply 🎯 𝗥𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱𝗲𝗱 𝗘𝗞𝗦 𝗔𝗱𝗱-𝗼𝗻𝘀 🔸 AWS Load Balancer Controller 🔸 Metrics Server 🔸 Cluster Autoscaler 🔸 External DNS 🔸 EBS CSI Driver 🔸 Prometheus & Grafana 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗮𝘀 𝗰𝗼𝗱𝗲 𝘄𝗶𝘁𝗵 𝗧𝗲𝗿𝗿𝗮𝗳𝗼𝗿𝗺 𝗺𝗮𝗸𝗲𝘀 𝗱𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁𝘀: ✔ Repeatable ✔ Scalable ✔ Version Controlled ✔ Automated 📌 𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆: Automate Everything. Document Everything. Infrastructure as Code is the foundation of modern cloud-native deployments. #AWS #EKS #Terraform #Kubernetes #DevOps #CloudComputing #InfrastructureAsCode #IaC #PlatformEngineering #CloudNative #Containers #Docker #AmazonWebServices #K8s #DevOpsEngineer #SRE #Automation #AnushaJanjirala

Production Kubernetes cluster is down. Your manager is asking for updates every 5 minutes. Here’s your step-by-step troubleshooting playbook: Step 1: Get your bearings Check where you are: kubectl config current-context See all contexts: kubectl config get-contexts Switch if needed: kubectl config use-context name List namespaces: kubectl get ns Step 2: See the big picture Node health: kubectl get nodes All pods: kubectl get pods -A Recent events: kubectl get events –sort-by=.metadata.creationTimestamp -A This tells you if it’s a cluster-wide issue or isolated problem. Step 3: Focus on the failing pod Get details: kubectl describe pod podname -n namespace Check logs: kubectl logs podname -n namespace Get inside: kubectl exec -it podname -n namespace – /bin/sh Step 4: Check health probes Look for probe failures in the describe output Test probe endpoint: kubectl exec -it podname -n namespace – curl localhost:port/health Step 5: Check deployments and rollouts Rollout status: kubectl rollout status deployment/name -n namespace View history: kubectl rollout history deployment/name -n namespace Rollback: kubectl rollout undo deployment/name -n namespace Step 6: Verify networking List services: kubectl get svc -n namespace Check endpoints: kubectl get endpoints -n namespace Test DNS: kubectl exec -it podname – nslookup servicename Step 7: Quick fixes that work Restart deployment: kubectl rollout restart deployment/name -n namespace Delete problematic pod: kubectl delete pod podname -n namespace The key is following the steps in order, not jumping around randomly.

Everybody says not to start with Kubernetes but to paraphrase Kelsey Hightower, "Kubernetes is the new Linux", so it's important to know how to build on it. Here's how I would build a future-proof Kubernetes cluster on AWS and GitHub that will allow you to iterate quickly and scale up when you need to. I like to think of Kubernetes-based infrastructure in three layers. They need to be deployed in this order: 1) Physical infrastructure layer (a set of nodes running Kubernetes) 2) Application infrastructure layer (the services you need to run your apps, things like: cert managers, ingress controllers, etc) 3) Application layer (your actual apps you want to run) Here's the high-level recipe: 1) Start by deploying AWS EKS and its requisite resources with Terraform. This should be a single Terraform module in a GitHub repo with a GitHub Action for automatically running `terraform apply` when you push a commit to main. 2) Use Terraform to bootstrap FluxCD on your cluster to allow you to run "GitOps". This can be in the same module as the above just add the right dependencies. 3) Store your application infrastructure services as Helm charts in a GitHub repo. Use FluxCD HelmRelease resources to automatically sync these charts to your cluster (this is the GitOps part). This should include things like AWS External Secrets Operator, Nginx Ingress Controller, etc. 4) Containerize your app components and add GitHub Actions to their repos to build and publish images to ECR when you commit to main. 5) Write Helm charts for each of your app's components (frontend, backend, queue, DB, etc) and add HelmRelease resources for them. 6) Add ImageUpdateAutomation resources to scan ECR and update your Helm chart values for each of your app components. This will trigger a deployment from the HelmRelease setup we did in step 5. This can be a lot of work and there are lots of details but the end result is: - To deploy your latest code to the cluster all you have to do is commit to the main branch. - To update a Helm chart all you have to do is commit to the main branch and bump the version tag in the HelmRelease resource. - To scale up your apps just edit one value in their HelmRelease. - To scale up your cluster just change one number in the Terraform values. - To debug your system just use standard `kubectl` commands that AI knows well. - All of this infrastructure and configuration is well versioned and centralized in a single repo so you can add branch protections and prevent drift. This might be overkill for your vibecoded todo app but for any system requiring serious infrastructure considerations I'd argue its better to pay this cost upfront before your customers come knocking about downtime and latency issues later.