At 03:14 on a Tuesday, a payment service in a mid-sized European fintech began returning HTTP 200 with an empty body. The checkout flow interpreted this as success. Balances were not updated. For eleven minutes, customers saw “order confirmed” while no money moved. The correctness suite had passed in CI forty minutes earlier. The resilience suite did not exist.
This is the gap this article examines. Correctness testing asks whether the system produces the right output for a given input under assumed conditions. Resilience testing asks whether the system continues to deliver acceptable behavior when those assumptions break: when a dependency times out, when a queue backs up, when a feature flag flips mid-request, when a downstream service returns a malformed 200. For senior practitioners, the distinction is not academic. It determines which failures you find before customers do, and which failures you only find in a postmortem.
Adjacent concepts matter here: fault injection, chaos engineering, degraded-mode operation, contract testing, property-based testing, and observability-driven development. Each has a different relationship to the correctness/resilience split. This article treats test suites, dashboards, and incident timelines as evidence to be cross-examined, not rituals to be performed.
What Correctness Testing Actually Proves
Correctness testing is the default mode of most quality engineering. Unit tests verify that a function maps inputs to outputs. Integration tests verify that components agree on interfaces. End-to-end tests verify that a user journey completes under a known data set. The underlying assumption is that the environment is stable enough to be specified.
This is not a weakness. Correctness testing is the only reliable way to catch regressions in business logic, calculation errors, and contract mismatches. A well-maintained correctness suite can tell you, with reasonable confidence, that the system does what the specification says when the specification holds.
But the specification rarely includes the failure modes that matter in production. A test that asserts a 200 response does not distinguish between a 200 with a valid body and a 200 with an empty body. A test that mocks a database does not exercise connection pool exhaustion. A test that runs in a clean environment does not see the effect of a noisy neighbor on the same host.
In the fintech incident above, the correctness suite passed because the mocked payment gateway returned a well-formed response. The production gateway returned a well-formed HTTP status with a malformed body. The test suite had no way to express “response shape is part of the contract.”
What Resilience Testing Actually Proves
Resilience testing starts from a different premise: the system will be operated in conditions you did not anticipate. The goal is not to verify a specification but to discover how the system degrades when the specification is violated.
Common techniques include:
- Fault injection: deliberately introducing latency, packet loss, or malformed responses into a dependency.
- Chaos engineering: terminating instances, saturating CPU, or partitioning the network in a controlled environment.
- Property-based testing: generating thousands of unusual inputs and asserting invariants that must hold for all of them.
- Contract testing with negative cases: verifying that a consumer handles a producer’s error responses, not just its happy path.
- Load and soak testing: observing behavior under sustained pressure, where resource leaks and queue buildup emerge.
Resilience testing does not replace correctness testing. It answers a different question. A correctness test asks, “Does this function return the right value?” A resilience test asks, “What happens when this function is called 10,000 times in a row with a 2-second delay on every third call?”
The Empty 200: A Field-Observed Failure Mode
I have seen this pattern in three separate organizations. A service returns HTTP 200 with an empty body, a null JSON field, or a truncated payload. The consumer treats the status code as the contract. The producer treats the body as optional. No test catches the mismatch because both sides are tested in isolation with mocks that assume the other side is well-behaved.
This is a resilience failure, not a correctness failure. The system is correct under the assumed contract. It is not resilient to a contract violation. The fix is not a better unit test. It is a contract test that includes negative cases, plus a consumer-side validation layer that rejects malformed responses before they propagate.
One team I worked with added a single property-based test that generated random JSON shapes for a downstream response. The test failed within minutes, revealing that the consumer crashed on a missing field that the producer’s documentation listed as optional. The correctness suite had been green for months.
Why the Distinction Gets Blurred in Practice
Many teams treat resilience testing as a subset of correctness testing. They add a few timeout tests, label the suite “resilience,” and move on. This misses the point. Resilience testing is not about adding more test cases. It is about changing the question you ask.
A correctness test has a known expected output. A resilience test often has no single expected output. Instead, it has invariants: the system must not crash, must not corrupt data, must not return a success response for a failed operation, must not exceed a latency budget under degraded conditions. These invariants are properties, not examples.
This is why property-based testing is a natural fit for resilience work. Instead of writing “given input X, expect output Y,” you write “for any input in this generator, the system must satisfy invariant Z.” The generator produces the edge cases you would not think to write by hand.
Operational Honesty and the Test Suite
This blog’s first pillar is operational honesty. A test suite that passes while production fails is not honest. It is a dashboard that lies. The lie is usually unintentional: the suite was built to verify a specification, and the specification did not include the failure mode that actually occurred.
Operational honesty requires acknowledging what your tests do not cover. A green CI pipeline is not evidence that the system is resilient. It is evidence that the system is correct under the conditions you thought to test. The gap between those two statements is where incidents live.
One practical step: add a resilience coverage report to your postmortem template. For each incident, ask which resilience property was violated and whether any existing test would have caught it. If not, the test suite has a blind spot. Document it. Do not let the postmortem end with “we added a test for this specific case.” That is correctness thinking applied to a resilience problem.
Broken-Build Sociology: Who Owns the Resilience Test?
The second pillar of this publication is broken-build sociology. Who is responsible when a resilience test fails? The answer is often unclear, and that ambiguity is itself a finding.
Correctness tests usually have a clear owner: the developer who wrote the code. Resilience tests often span team boundaries. A fault injection test that kills a database instance may fail because of how the application handles the connection loss, how the infrastructure re-routes traffic, or how the database cluster elects a new primary. No single team owns the whole path.
This is why resilience testing requires a different social contract. The test is not a gate that blocks a merge. It is a signal that the system’s degradation behavior has changed. The response is not “fix your code” but “let’s look at the evidence together.” Teams that treat resilience test failures as blame events stop running them. Teams that treat them as shared observations keep running them.
Beta Feedback Contracts and Resilience Signals
The fourth pillar of this publication is beta feedback contracts. Beta programs are often used to validate correctness: does the feature work as designed? But beta users are also a source of resilience signals, if you ask the right questions.
Instead of asking “Did the feature work?” ask “What happened when your network dropped mid-action?” or “Did you ever see a success message that turned out to be wrong?” These questions surface the empty-200 class of failures that automated tests miss. They also set expectations: beta users are not just feature validators. They are degradation observers.
One beta program I reviewed collected 47 bug reports. Forty-one were correctness issues. Six were resilience issues: partial saves, stale confirmations, and one case where the app showed a cached success after the server had rejected the request. The six resilience reports led to more architectural change than the forty-one correctness reports combined.
Counterarguments and Tradeoffs
Resilience testing has real costs. Fault injection in production requires careful blast-radius control. Property-based tests can be slow and flaky if generators are not tuned. Chaos engineering can surface so many issues that teams drown in findings. And not every system needs the same level of resilience investment. A batch job that runs nightly and can be re-run has different requirements than a payment gateway.
The counterargument to resilience testing is usually: “We don’t have time to test for failures we haven’t seen.” This is a reasonable position for a small team with a simple system. It becomes less reasonable as the system grows, as dependencies multiply, and as the cost of an incident rises. The tradeoff is not between testing and not testing. It is between finding degradation in a controlled environment and finding it in a customer-facing incident.
There is also a risk of over-rotating. A team that invests heavily in chaos engineering while its correctness suite is weak will find many ways the system fails, but will not know whether the system works at all. The two disciplines are complementary, not competing. The right balance depends on the system’s failure history, not on a generic best practice.
What to Do Next: Non-Obvious Recommendations
1. Add a “Malformed Success” Test to Every Consumer Contract
For every external dependency, write one test that returns HTTP 200 with an empty body, one with a truncated JSON payload, and one with a null required field. Assert that the consumer rejects the response and logs a structured error. This single practice catches the empty-200 class of failure that correctness suites miss.
2. Run a Quarterly “Assumption Audit” on Your Test Suite
List every assumption your tests make about the environment: database is up, network is fast, third-party API returns valid JSON, clock is monotonic. For each assumption, ask what happens when it breaks. If you cannot answer, you have a resilience blind spot. Document it in the same place you document known bugs.
3. Separate Correctness Gates from Resilience Signals
Do not block merges on resilience tests that are inherently flaky or environment-dependent. Instead, run them on a schedule and route failures to a shared channel. The goal is not to stop a bad merge. It is to build a shared picture of how the system degrades. This social change is often harder than the technical change, and more valuable.
4. Add a “Resilience Debt” Section to Your Postmortem Template
For every incident, record which resilience property was violated, whether any existing test would have caught it, and what new test or observability signal would have shortened detection. Review this section quarterly. Patterns will emerge that point to structural weaknesses, not just one-off bugs.
Frequently Asked Questions
What is the difference between correctness testing and resilience testing?
Correctness testing verifies that the system produces the right output for a given input under assumed conditions. Resilience testing verifies that the system continues to deliver acceptable behavior when those assumptions break, such as when a dependency times out, returns malformed data, or disappears entirely. Correctness tests have known expected outputs; resilience tests often assert invariants that must hold across many unusual inputs.
Can a test suite be both correct and not resilient?
Yes. A suite can pass every correctness test while the system fails in production under degraded conditions. The fintech example in this article is a case in point: the mocked payment gateway returned a well-formed response, so the suite passed. The production gateway returned HTTP 200 with an empty body, which the consumer treated as success. The suite was correct under the assumed contract but not resilient to a contract violation.
What is the “empty 200” failure mode?
The empty 200 is a failure mode where a service returns HTTP 200 with an empty body, a null field, or a truncated payload, and the consumer treats the status code as the entire contract. The operation appears to succeed but no work is done. This class of failure is invisible to most correctness suites because mocks assume well-formed responses. It is a resilience failure, not a correctness failure.
How do I start resilience testing without a dedicated chaos engineering team?
Start with negative contract tests: for each external dependency, add tests that return malformed success responses and assert that the consumer rejects them. Then add a small set of property-based tests that generate unusual inputs and assert invariants. Run these on a schedule rather than as merge gates. The goal is to build a shared picture of degradation behavior, not to block every merge.
Where This Leads
This article is the first in a planned series on testing epistemology. The next piece will examine contract testing as a lie detector: how consumer-driven contracts expose the assumptions that integration tests hide. If you have a field story about a resilience failure that a correctness suite missed, the comments are open. The evidence is the point.
