Question

How do you patch and update production systems without impacting uptime?

With care, patches in a production system should not have downtime. You should outline your process on how to test patches in staging and use blue-green or canary deployments to schedule patches during low-traffic times. This is where the tools such as Ansible or Chef help, allowing automation to do the least amount of disruption possible while patching.

Answer 1

Rolling updates: I roll out production instances one by one using rolling update strategies to maintain overall uptime. Kubernetes manages rolling updates at the pod level, thus updating only a portion of the system at any one time.
Blue-Green and Canary Deployments: In a blue-green deployment, the new version is deployed in a parallel environment, and traffic is switched over once the new version is stable. With canary releases, updates are initially rolled out to a small set of users, and traffic is gradually increased as the new version performs well. These deployment strategies minimize user impact and simplify rollback if needed.
Automated Health Checks and Monitoring: Automated tests validate any change before and after deployment. Health checks in Kubernetes or load balancers, like AWS ELB, mean only healthy instances can be routed to, thereby letting us roll back a deployed version if a fault occurs after deployment.