Authority lesson · 810 words
Observability, Reliability, and Incident Response
Observe outcomes, not only components
Observability should help answer why a user-visible outcome failed. Start with critical journeys: sign in, submit an audit, receive a report, accept a proposal, or process a payment. For each journey, define success, latency, dependency, and correctness signals. CPU and memory matter, but a healthy server can still deliver no emails or leave every job queued. Connect request IDs, job IDs, tenant IDs, and provider message IDs so one transaction can be followed without logging private content.
Use structured logs for discrete events, metrics for aggregated rates and distributions, and traces for causal paths across boundaries. Avoid using logs as an unbounded database. Set retention, redaction, and access controls. Ensure errors include actionable context—operation, safe identifiers, retry state, and dependency—without secrets, tokens, full request bodies, or intimate user content.
Reliability targets force prioritisation
A service-level indicator is a measured behaviour such as successful audit completion within ten minutes. A service-level objective is the target, such as 99 per cent over thirty days. Choose indicators that reflect customer value and can be measured consistently. Separate availability from correctness; a report generated with missing pages may be technically complete but not useful. Define exclusions narrowly so maintenance or provider failures do not make the number meaningless.
Use an error budget to decide how much change risk is acceptable. If reliability is below target, reduce risky releases and invest in failure reduction. Do not market an uptime percentage that cannot be supported by monitored evidence and a defined measurement window. Public claims should match the actual service boundary and reporting method.
Alerts should lead to an action
Alert on conditions that require a human or automated response: sustained user-journey failure, growing queue age, repeated audit-delivery failures, exhausted database connections, or backup verification failure. Avoid paging on every exception. Include runbook links, scope, recent changes, and safe diagnostic queries. Use severity to distinguish immediate customer or security risk from work that can wait for business hours.
Test alerts by deliberately triggering them in staging. A rule that has never fired may reference the wrong label or route to an abandoned channel. Review noisy alerts after incidents and quiet periods. Silence must have ownership and expiry; otherwise temporary suppression becomes permanent blindness.
Incident response is a decision system
Prepare roles for incident lead, operations, communications, security, and record keeping. The first goals are safety, scope, containment, and preservation of evidence—not instant root cause. Use a timeline with facts and confidence levels. Separate customer communication from speculative technical discussion. Define when to disable a feature, stop a worker, revoke credentials, restore data, or shift traffic.
After stabilisation, verify recovery through user journeys and data checks. Conduct a blameless review that still names control failures, decisions, and owners. Correct the conditions that allowed the incident: missing limits, untested rollback, weak observability, unclear ownership, or unsafe defaults. Track actions to completion and verify that the new control works.
Scenario
Audit jobs complete, but customers receive nothing
The web process, database, and worker all report healthy. A Mailgun configuration error causes every customer delivery to fail, but the application marks reports as notified. Component monitoring misses the incident because no user-outcome signal exists. Separate delivery records, provider IDs, bounded retries, and an alert on failed or stale customer delivery reveal the actual failure.
- Define customer delivery as a journey
- Record provider evidence separately from job completion
- Alert on failed and stale deliveries
- Verify recovery by receiving a real message
Worked example
Instrument a free-audit journey
- Assign one correlation ID at submission.
- Record queue wait, crawl duration, report generation, customer delivery, and internal delivery as separate events.
- Emit success and latency metrics for each stage.
- Alert when customer delivery remains failed beyond the retry policy.
- Use a synthetic domain and mailbox in staging to verify the complete path.
Checklist
Use this before acting
- Define user journeys and success criteria
- Correlate requests, jobs, tenants, and provider messages
- Redact secrets and sensitive content
- Set measurable reliability objectives
- Alert only when an action exists
- Test alert routing and runbooks
- Prepare incident roles and containment decisions
- Verify recovery through real journeys
- Complete and verify corrective actions
Common mistakes
Failure patterns to avoid
- Monitoring only host health
- Treating all HTTP 200 responses as success
- Logging raw sensitive payloads
- Publishing unsupported uptime claims
- Paging on every exception
- Declaring recovery before testing the customer journey
Practical exercise
Create one reliability and incident pack
Choose a critical journey. Define indicators, objective, error budget, structured events, dashboard, two actionable alerts, a staging failure injection, and an incident runbook with containment and verification.
Deliverable: A dashboard specification, tested alert, and incident-response runbook.
Sources and further reading
Primary and official references
Topic-specific takeaway
Reliability becomes manageable when user outcomes are measurable, alerts have actions, and incident decisions are rehearsed and verified.
