Another incident report landed on my desk. Same story. Monitoring flagged a spike in p99 latency. The on-call engineer got the page, then burned forty minutes restarting services, cycling connection pools, and finally rolling back a release that had been running fine for six hours. The rollback “fixed” it. The postmortem will call it a successful detection. I call it something else: a test someone never wrote.
Over the last decade, a quiet assumption has settled into engineering orgs. It whispers that if we pile up enough dashboards, alerts, and anomaly detectors, we can ease off on testing. The system will tell us when it breaks. We’ll fix it fast. MTTR will drop. The problem? This confuses noticing a failure with stopping a failure. Those are not the same job. They don’t carry the same cost. And the gap between them is exactly where your most painful outages take root.
The Detection Trap
Let’s get precise. Monitoring tells you something already went sideways. It’s reactive by design. A latency alert fires because requests are already timing out. A memory threshold trips because a GC storm is already chewing through your heap. An error-rate spike means customers are already staring at 500s. Monitoring has real value—it shrinks the gap between the start of a problem and the moment a human knows about it. But shrinking that window does zero to stop the problem from lighting up in the first place.
Testing, on the other hand, is proactive. A solid integration test walks a code path under controlled conditions before that path ever sees real traffic. A load test pushes a subsystem to its breaking point in a sandbox, not at 3 a.m. on a Saturday. A chaos experiment injects failure on purpose to watch the system’s response, with no customer stuck on the other end. The difference sits in time: testing happens before the event; monitoring happens after.
When a team starts leaning on monitoring as a stand-in for testing, they quietly accept that failures will reach users. The number they end up optimizing isn’t reliability—it’s time-to-acknowledge. That metric feels seductive because it looks like control. But it’s just control over the cleanup, not over the mess.
What Monitoring Can’t See
Even the fanciest observability stack has hard limits. I’ve watched teams sink months into tuning Prometheus alerts, only to get blindsided by a failure mode their metrics never covered. The reason is structural: monitoring only sees what you explicitly wired up. If you never thought to measure connection pool saturation in that dusty legacy service nobody touches, you won’t get an alert when it silently starves its threads. If you didn’t instrument the exact error code coming back from a third-party API, your dashboard will flash “external service failure”—and you won’t know it’s because your auth token format shifted in a way that sails through unit tests but fails the real handshake.
Testing forces you to list out failure modes. A decent test suite doesn’t just verify the happy path. It pokes at the ugly questions: What happens when this dependency burps back a malformed response? What happens when the database connection hangs for 30 seconds? What happens when the message queue doubles a message? These questions turn into assertions. Monitoring, meanwhile, sits back and waits for nature to run the experiment for you—and nature makes for a cruel experimentalist.
The Cost Economics Are Backward
There’s a tired line that testing is expensive and monitoring is cheap. Writing tests eats developer time. Maintaining test infrastructure takes effort. Running a full suite in CI slows the pipeline. A monitoring SaaS subscription, by comparison, is just a line item. A few dashboard panels and some alert rules, and you’re done. This math is wrong in a very specific way: it ignores the cost of unplanned work.
When an alert fires in production, the cost isn’t just the on-call engineer’s time spent diagnosing. It’s the context-switch tax on every engineer dragged onto the bridge. It’s the delay to the feature work that was supposed to ship that sprint. It’s the reputational sting if customers noticed. It’s the compounding cost of a rushed fix that introduces a new, sneakier bug because the team was under pressure to restore service. These costs are diffuse, slippery to measure, and almost never show up on the monitoring vendor’s ROI calculator.
A test that catches a regression before merge carries a laughably lower cost profile. The developer is already in the code. The fix takes minutes. Nobody gets paged. No status page updates. No postmortem to write. The trade-off isn’t “expensive testing” vs. “cheap monitoring.” It’s predictable, low-cost prevention vs. unpredictable, high-cost reaction.
The Alert Fatigue Multiplier
There’s a secondary cost teams ignore until it’s too late. When monitoring serves as the primary safety net, alert volume tends to creep up. Every near-miss spawns a new alert rule. Every postmortem action item includes “add alerting for this condition.” Over time, the signal-to-noise ratio craters. Engineers start muting channels. They develop a learned helplessness toward the pager. When a genuinely critical alert fires, the response is slower—not because the engineer is careless, but because a hundred false alarms have conditioned them to hesitate.
Testing shrinks alert volume by wiping out whole classes of failures. If you have a test that verifies the new caching layer handles eviction correctly, you don’t need an alert to scream about it in production—because it won’t be broken. Every test is a silent, automatic dismissal of a future alert. The dashboard gets quieter. The on-call rotation becomes less punishing. This is a systemic improvement that monitoring alone can’t deliver.
The False Dichotomy of Staging Environments
A common pushback I hear: “We test in staging, but production is different. We need monitoring because we can’t replicate everything.” That’s true, as far as it goes. Production has traffic patterns, data shapes, and infrastructure quirks no staging environment perfectly mirrors. But it’s a mistake to conclude testing is therefore pointless. It’s a mistake to let the perfect become the enemy of the good enough to catch 80% of regressions.
The right answer to “staging isn’t production” isn’t to abandon staging. It’s to supplement it with production-like testing techniques. Canary deployments are a form of testing. Traffic shadowing is a form of testing. Feature flags with gradual rollouts are a form of testing. These techniques shove the testing boundary closer to production without crossing the line into “we’ll just stare at the dashboards and hope.” They acknowledge the gap between environments and actively close it, rather than using it as an excuse to stop trying.
The Culture Shift Required
Making this change demands something harder than buying a tool or writing a runbook. It requires a cultural commitment that makes many engineering managers squirm. You have to slow down merges to speed up delivery. You have to invest in test infrastructure when no fire is burning. You have to praise the engineer who wrote a test that prevented an outage that didn’t happen—which is a brutally hard thing to celebrate because its success is invisible.
The orgs that get this right share a few traits. They treat test coverage gaps with the same severity as production incidents. They include test quality in code review standards, not as a checkbox but as a substantive discussion. They invest in test data management and environment provisioning so that running a comprehensive suite is fast and reliable. And they never, ever accept “we’ll catch it in monitoring” as an excuse for skipping a test.
Practical Steps to Rebalance
If you’re nodding along but wondering where to start, here are concrete moves that don’t demand a reorg or a massive budget:
1. Audit your last five production incidents. For each one, ask: could a test have caught this? Be specific. Was it a missing unit test on a boundary condition? A missing integration test on a contract change? A missing load test on a new endpoint? If the answer is yes, write that test. Don’t just add an alert. The alert is the backup; the test is the primary.
2. Identify your alerting “greatest hits.” Look at the alerts that fire most often. For each, ask: is there a code or infrastructure change that would make this alert unnecessary? A retry with backoff? A circuit breaker? A fallback path? Implement the fix and remove the alert. An alert that never fires is a solved problem; an alert that fires constantly is a cry for engineering attention.
3. Run a pre-production checklist for every release. This isn’t a manual form. It’s an automated gate. Before a release candidate can go to production, it must pass: the full unit and integration suite, a smoke test against a production-like environment, and a targeted load test for any changed paths. If any of these fail, the release is blocked. This makes testing a gate, not a suggestion.
4. Practice failure injection during working hours. Use something simple. Kill a pod. Sever a network link. Corrupt a message. Watch what happens. If your monitoring catches it before a human notices the degradation, you have an acceptable safety net. If a customer would have seen an error, you have a testing gap. Close it.
The Overlap That Matters
I’m not arguing monitoring is worthless. I’m arguing it’s a different tool for a different job. Monitoring excels at detecting the unknown unknowns—the emergent behaviors no test suite could have predicted. A memory leak that only shows up after 14 days of uptime. A race condition triggered by a specific interleaving of traffic nobody thought to simulate. These are the failures that make distributed systems genuinely hard, and monitoring is essential for catching them.
But the known unknowns—the failure modes you can anticipate, the edge cases you can list, the contracts you can verify—those belong to testing. When you shove them onto the monitoring system, you’re not being pragmatic. You’re being lazy. You’re trading a small bit of engineering time now for a large pile of engineering time (and customer pain) later.
The strongest engineering cultures treat testing and monitoring as complementary layers of a defense-in-depth strategy. Testing catches everything it can before release. Monitoring catches everything that slips through. The goal is to make the monitoring layer as boring as possible. A quiet dashboard signals engineering maturity, not that monitoring is unnecessary.
Frequently Asked Questions
Isn’t it impossible to test everything? Doesn’t that mean monitoring is necessary?
Yes, testing everything exhaustively is impossible. And yes, monitoring is necessary. The point isn’t to eliminate monitoring—it’s to stop using monitoring as an excuse to skip the tests you can write. Many teams have a large gap between what they could reasonably test and what they actually test. Closing that gap reduces the burden on monitoring and makes the system genuinely more reliable.
We have a small team and limited time. How do we prioritize testing?
Focus your testing efforts on the paths that have caused the most pain historically. Look at your incident record. The areas that break most often are the areas that will benefit most from automated tests. Also prioritize any path that involves money, personal data, or irrecoverable operations. A cheap smoke test that prevents one data-loss incident can justify months of testing investment.
Can’t we just use canary deployments and feature flags instead of writing tests?
Canary deployments and feature flags are testing techniques—they’re just pushed closer to production. They are valuable, but they still operate on the “detect and rollback” model. A problem caught by a canary is still a problem that reached real users, even if only a small percentage. A pre-production test catches it before any user is affected. Use canaries as a safety net, not as a replacement for earlier testing stages.
How do I convince management that testing is worth the investment when monitoring seems cheaper?
Track the cost of unplanned work. For each production incident, log the total engineering-hours spent on diagnosis, remediation, and postmortem follow-up. Include the opportunity cost of delayed feature work. Compare that number to the cost of the testing that would have prevented the incident. In most organizations, the math favors testing by a wide margin once you measure it honestly. Present the data, not a philosophical argument.
The next time you’re about to merge a pull request and you hear yourself think, “We’ll catch it in monitoring if something goes wrong,” pause. Ask yourself what you’re really saying. You’re saying you’re willing to let a customer find this bug. You’re willing to wake someone up at 2 a.m. You’re saying the test isn’t worth the time. Maybe it isn’t—but be honest about the trade-off. Monitoring is not a substitute for testing. It’s a confession that testing wasn’t done.