Authority lesson · 881 words
HTTP Clients, Service Boundaries, and Resilience
Treat the remote service as an unreliable boundary
An HTTP call is not a function call that happens to use a URL. It crosses networks, processes, organisations, authentication systems, rate limits, and deployment schedules. The request may reach the remote service even when your client times out. A response may be syntactically valid but semantically incomplete. The remote API may change behaviour without changing its status code. Design the boundary so the rest of the application does not need to understand every transport detail.
Create a small client interface expressed in domain language: create_invoice, fetch_monitor_status, or submit_report. Keep URL construction, headers, authentication, serialization, and response parsing inside the adapter. Return typed results or raise a narrow set of meaningful exceptions such as temporary_unavailable, rejected_request, authentication_failed, and invalid_response. This stops HTTP vocabulary from leaking into business policy and makes the failure choices testable.
Timeouts are part of correctness
Every outbound request needs explicit connection and read timeouts. A connection timeout limits how long the client waits to establish a network path; a read timeout limits silence after connection. One generous combined timeout can trap worker capacity during partial failure. Choose values from the user-facing or job-level deadline, not from habit. If a request is part of a ten-second web response, a thirty-second default is already incorrect.
Propagate a deadline when one operation makes several calls. Each retry consumes time, so calculate whether another attempt can finish before that deadline. When the operation is asynchronous, record the attempt and schedule a later retry rather than sleeping inside a worker indefinitely. Make timeout outcomes visible to operations instead of hiding them behind a generic error.
Retry only operations that can be repeated safely
Retrying a GET is usually less dangerous than retrying a charge, reboot, invitation, or resource-creation request. Even safe reads can amplify an outage if thousands of clients retry together. Use exponential backoff, a maximum attempt count, and jitter. Respect Retry-After when the service provides it. Avoid retrying authentication failures, validation failures, and other responses that require changed input.
For state-changing requests, use an idempotency key supported by the provider or create an internal operation record before sending. The key remains stable across retries of the same intent and changes for a genuinely new intent. Persist the provider request or resource identifier so reconciliation can distinguish “not sent,” “sent but response lost,” and “confirmed.” Never equate a timeout with proof that nothing happened.
Resilience includes observability and containment
A client should emit enough evidence to diagnose failure without leaking credentials or private payloads. Record provider, operation, status category, attempt number, duration, correlation identifier, and a safe error code. Do not log authorization headers, API keys, full personal records, or arbitrary response bodies. Metrics should distinguish latency, rejection, throttling, timeout, and invalid-response failures because each requires a different response.
Add a circuit breaker or temporary suppression rule when repeated failures would only create load. A breaker is a policy describing which failures count, how long the boundary remains open, what callers receive, and how recovery is tested. Provide a degraded path where possible—queue the work, show stale-but-labelled data, or let staff retry after review—rather than pretending the remote dependency is always available.
Scenario
A timeout creates two customer accounts
A portal posts to a provider and waits eight seconds. The provider creates the account at second seven, but the response is lost. The portal treats the timeout as “nothing happened” and retries with a new request identifier. Two external accounts are created, and reconciliation cannot tell which belongs to the customer. The defect is a missing operation identity, not merely a timeout setting.
- Create an internal operation before the first request
- Use one idempotency key across retries
- Store the external ID when any response arrives
- Move uncertain outcomes to reconciliation instead of blind retry
Worked example
Define a resilient report-submission client
- Expose submit_report(report_id, artifact_hash) rather than a generic post method.
- Use separate connection and read timeouts plus a total deadline.
- Persist an operation record with idempotency key, request hash, attempts, status, and external identifier.
- Retry only timeout, reset, 429, and selected 5xx outcomes with jitter.
- Translate provider responses into accepted, rejected, uncertain, or temporary_failure outcomes.
Checklist
Use this before acting
- Use a domain-specific client interface
- Set explicit connect and read timeouts
- Define retryable and non-retryable outcomes
- Use stable idempotency keys for state changes
- Validate response shape and semantics
- Redact secrets and private payloads from logs
- Record correlation IDs, duration, attempts, and provider status
- Provide reconciliation for uncertain outcomes
Common mistakes
Failure patterns to avoid
- Calling requests without a timeout
- Retrying every exception
- Generating a new idempotency key on each attempt
- Returning raw provider dictionaries throughout the application
- Logging entire error responses
- Assuming a timeout proves the provider did nothing
Practical exercise
Harden one external integration
Choose an HTTP integration. Draw its boundary, list every outcome the caller must handle, and identify state-changing requests. Add timeouts, a stable operation identifier, redacted structured logging, and tests for timeout-before-send, timeout-after-send, throttling, invalid JSON, and reconciliation.
Deliverable: A client contract, retry table, operation-state diagram, and boundary tests.
Sources and further reading
Primary and official references
Topic-specific takeaway
Resilient HTTP code preserves the identity and state of an operation even when the network cannot tell you exactly what happened.
