Authority lesson · 1066 words
Testing Strategy, Fixtures, Fakes, and Property Thinking
A strategy begins with the failure you cannot afford
A test strategy is not a list of tools. It is an argument about which failures matter, where those failures can be detected most cheaply, and what evidence a reviewer needs before approving a release. Begin with the system’s promises: accepted inputs, rejected inputs, state changes, external calls, security boundaries, and recovery behaviour. Rank the damage caused by each promise failing. A calculation error in an internal report may be inconvenient; a tenant-boundary failure or duplicate payment can be catastrophic. The highest-consequence promises deserve tests at more than one level, while low-risk formatting details may need only a focused unit test or visual review.
This risk-first approach avoids two extremes. One is a suite made almost entirely of tiny unit tests that prove isolated methods while missing broken integrations. The other is a slow end-to-end suite that reproduces the whole system for every assertion and becomes too expensive to trust. A balanced suite uses the smallest boundary that can expose the relevant failure, then adds a limited number of broader checks to prove that database, queue, browser, and provider boundaries connect correctly.
Use fixtures to express state, not hide it
Fixtures should make the important starting conditions obvious. A useful fixture names the business state it creates: an active organisation with an expired subscription, a report with two invalid rows, or a user lacking the required role. A harmful fixture creates dozens of unrelated objects through global setup, leaving the reader to guess which fields matter. Prefer small factories or builders with safe defaults and explicit overrides. Keep time, randomness, filesystem paths, and environment variables under the test’s control so the same scenario is reproducible on a clean machine.
Treat fixture difficulty as design feedback. If creating one valid object requires twenty unrelated arguments, the production model may carry too many responsibilities. If every test patches five globals, dependency boundaries are probably hidden. Tests are not only verification; they expose where construction, policy, and infrastructure have become tangled. Repairing the design often makes both the tests and the application easier to reason about.
Choose fakes and mocks deliberately
A fake is a working, simplified collaborator: an in-memory repository, deterministic clock, or mail transport that records messages. A mock is an expectation about an interaction. Fakes are often better for behavioural tests because they permit realistic sequences without coupling the test to every internal call. Mocks are valuable when the interaction itself is the contract—for example, proving that a recovery action is never invoked without approval, or that a payment request carries the required idempotency key.
Avoid mocking code you own simply to make a test pass. When every internal method is mocked, harmless refactoring breaks the suite even though behaviour remains correct. Mock at architectural boundaries: network clients, queues, clocks, randomness, operating-system calls, and third-party SDKs. Then assert business outcomes and only the safety-critical interactions. Run selected integration tests against the real database and real serialization rules because a fake cannot reproduce every constraint or transaction edge.
Property thinking extends beyond selected examples
Example tests answer, “What happens for these values?” Property thinking asks, “What must remain true across a broad class of values?” A parser may need to preserve row order, reject impossible dates, and never create more output records than valid input records. A pricing function may need to stay non-negative and monotonic. Express these properties with generated data, loops over carefully chosen partitions, or a property-testing library. The important step is identifying the invariant before generating values.
Generated tests do not replace named examples. Keep concrete cases for regulatory rules, prior incidents, and boundary values that humans must understand. Use property tests to explore combinations people are unlikely to enumerate, then retain any discovered failure as a readable regression test. Record the seed or minimised example so a failure can be reproduced rather than dismissed as random.
Scenario
The suite is green, but duplicate jobs are billed twice
A background import job is retried after a worker timeout. Unit tests prove the billing function calculates the correct amount, and controller tests prove a successful request returns 202. No test models the same job identifier arriving twice. In production, the retry creates a second charge. The missing test was not “more coverage”; it was a risk-based property: processing the same idempotency key repeatedly must produce one billable outcome.
- Identify the invariant before selecting the test layer
- Use a fake repository that enforces unique idempotency keys
- Add an integration test against the real database constraint
- Keep the incident as a named regression case
Worked example
Build a verification matrix for an import service
- List promises: reject malformed rows, preserve valid rows, avoid duplicates, record audit evidence, and fail safely when storage is unavailable.
- Assign boundaries: pure parser tests, service tests with a fake repository, database tests for uniqueness and transactions, and one end-to-end upload.
- Add properties: output never exceeds valid input; repeated import IDs do not create records; every rejected row has a reason.
- Record release evidence: command, environment, database version, fixture seed, result count, and regression links.
Checklist
Use this before acting
- Name the business risk each important test addresses
- Keep fixtures small, deterministic, and explicit
- Mock external boundaries rather than internal implementation details
- Test database constraints with the production database engine
- Include hostile, empty, duplicate, and interrupted inputs
- Preserve discovered failures as readable regressions
- Record the environment and command used for release evidence
Common mistakes
Failure patterns to avoid
- Measuring quality by coverage percentage alone
- Using one enormous fixture for every test
- Mocking every method and asserting implementation order
- Testing only on SQLite when production uses PostgreSQL
- Generating data before defining invariants
- Ignoring flaky tests instead of finding uncontrolled dependencies
Practical exercise
Write a risk-to-evidence plan
Choose a Python service. Write five promises and rank them by consequence and likelihood. For the top three, choose a unit, integration, contract, or end-to-end boundary and explain why it is the smallest useful boundary. Create one explicit fixture, one fake collaborator, one invariant, and one clean-environment release check.
Deliverable: A one-page verification matrix plus at least four executable tests and a release-evidence note.
Sources and further reading
Primary and official references
Topic-specific takeaway
A trustworthy test suite is a risk model expressed as executable evidence, not a contest to produce the largest number of assertions.
