Authority lesson · 805 words
Continuous Integration, Packaging, Deployment, and Operations
Continuous integration is an evidence pipeline
CI should answer whether a specific revision is suitable to become a release candidate. A useful pipeline checks syntax, formatting policy, tests, migration alignment, dependency resolution, security controls, package construction, and artifact integrity. It records the commands, environment, and outputs. A green result is meaningful only when protected branches, workflow review, and credential boundaries prevent an untrusted change from redefining the checks or publishing an artifact.
Order checks to fail cheaply first, but do not let quick checks replace production-relevant evidence. Unit tests may run on every commit, while PostgreSQL integration tests, browser tests, and clean-install rehearsals run before release. The release decision should point to all required evidence rather than assuming one job represents the whole system.
Build once and promote the artifact
A release should be built from a clean checkout with declared dependencies. Produce a wheel, container, or versioned source package once, calculate a checksum, and promote that same artifact through staging and production. Rebuilding separately in each environment creates different dependency resolution, timestamps, generated files, and opportunities for tampering. Record the source commit, dependency manifest, builder, artifact hash, and signing or storage controls.
Keep configuration and secrets outside the artifact. Validate required settings at startup without printing secret values. Include database migration boundaries, static assets, templates, and operational scripts in the manifest. Exclude caches, local databases, uploaded media, virtual environments, and developer secrets. An overwrite-safe installer should back up replaceable source, preserve runtime state, verify every payload member, and produce an installation receipt.
Deployment is a controlled state transition
Before deployment, verify backups and a disposable restore, migration compatibility, capacity, external dependencies, and rollback conditions. During deployment, apply changes in an order that preserves compatibility. After deployment, check process health, database state, representative user journeys, worker queues, email delivery, and observability. A process returning 200 on its root page is not enough evidence that audits, subscriptions, authentication, and background jobs work.
Rollback is not always “restore old files.” A migration or external side effect may make source rollback unsafe. Define whether rollback means previous artifact, forward fix, feature disablement, data restore, or traffic shift. Set the decision deadline before starting. Record who can make the call and which metrics trigger it.
Operate the release after the deploy command
Release evidence should continue into operations. Track deployed artifact hashes, migration versions, configuration revision, worker versions, and health checks. Alert on error rates, queue age, failed email delivery, stale jobs, and backup failures. Keep a changelog that names user-visible changes, data changes, known limitations, and support actions. Retain enough history to answer which customers and environments were exposed to a defective version.
Practice clean installation and upgrade from the oldest supported version. Verify rollback while preserving runtime files. Rotate credentials used by CI and deployment, restrict production access, and review dependencies regularly. Production readiness is not a one-time checklist; it is the repeatability of the build, deploy, verify, recover, and learn cycle.
Scenario
Production installs a dependency that staging never saw
Staging was deployed Monday from an unlocked requirements file. Production rebuilt Friday and resolved a newer transitive package with a regression. Both environments claim the same application version. Building one immutable artifact with a dependency manifest and checksum would make the difference visible and prevent environment-specific resolution.
- Resolve dependencies once in a clean build
- Promote the identical artifact
- Record hashes and environment receipts
- Verify representative user journeys after deployment
Worked example
Create a Python release evidence bundle
- Run tests and migration checks from a clean revision.
- Resolve dependencies and build one artifact.
- Create a manifest and SHA-256 checksum.
- Deploy the artifact to staging and run database, browser, worker, and email smoke tests.
- Promote the same artifact to production, verify, and record the receipt and rollback deadline.
Checklist
Use this before acting
- Protect CI workflows and publishing credentials
- Run production-relevant database and browser checks
- Build once from declared dependencies
- Checksum and retain the artifact manifest
- Preserve runtime state during source replacement
- Define migration and rollback conditions
- Verify representative journeys after deployment
- Record deployed versions and operational signals
Common mistakes
Failure patterns to avoid
- Treating a green unit-test job as complete release evidence
- Rebuilding separately in staging and production
- Packaging local databases or secrets
- Assuming file rollback reverses data changes
- Skipping clean-install and restore rehearsals
- Failing to record the deployed artifact hash
Practical exercise
Build a release-candidate evidence pack
Take one Python service from clean checkout to a versioned artifact. Include dependency resolution, tests, manifest, checksum, migration plan, staging verification, production verification, rollback criteria, and installation receipt.
Deliverable: A reproducible artifact plus an auditable release-evidence directory.
Sources and further reading
Primary and official references
Topic-specific takeaway
A production release is a traceable chain from reviewed source to one immutable artifact, verified deployment, observable operation, and rehearsed recovery.
