advanced

Rollbacks

Plan fast recovery with versioned artifacts, reversible migrations, health checks, and clear rollback ownership.

Rollback is a designed recovery path, not a panic button. It depends on immutable artifacts, compatible schema changes, traffic controls, observability, and clear ownership during incidents.

Recovery options include:

  • **Code rollback** — redeploy previous artifact version
  • **Config rollback** — revert feature flags or env settings
  • **Data repair** — forward-fix when schema cannot revert
  • **Traffic shift** — blue-green or canary reversal
					Trigger: error rate > SLO for 5m
Action: route traffic to previous revision + disable flag X
Owner: on-call + release captain
				

Practice restores before production incidents. Irreversible migrations and external side effects (payments, emails) may require forward-fix instead of revert.

On interviews: distinguish code, config, data, and flag rollback; when forward-fix is safer; rehearsed runbooks.

Common pitfalls: no immutable artifacts to return to; migrations that break old code paths; rollback without verifying health of the previous version.

The trade-off is keeping rollback simple versus shipping schema and API changes that complicate recovery.

Checklist:

  • Keep deploy artifacts immutable.
  • Use backward-compatible migrations.
  • Define rollback triggers and owners.
  • Rehearse recovery on staging.