There’s this quiet, stubborn idea that floats around engineering orgs. It goes like this: once you’ve got enough monitors wired up, you can finally relax on the testing front. The dashboards will glow red when something’s wrong. PagerDuty will scream. Someone will fix it. And all those slow, fiddly integration tests? They start to feel like a luxury—something you’ll get to when you’re not so busy shipping.
I’m going to be blunt. That idea isn’t just a little off. It’s wrong in a deep, structural way. Monitoring and testing aren’t two flavors of the same thing. They do entirely different jobs, on completely different timelines, and they guard against separate categories of disaster. Treating one like it can pinch-hit for the other is like confusing a smoke alarm with a building code. One tells you the fire is already eating the roof. The other stops the fire from starting.
The Temporal Wall
Testing happens before the code touches a real user. Monitoring happens after. That simple fact colors everything else.
Think about a test suite. You’re in a clean room. You pick the inputs. You declare the expected outputs. You wall the system off from any outside noise. When the test finishes, you get a single binary signal: pass, or fail. The whole point is that you learn something concrete about your code before it ever gets a chance to hurt someone.
Now look at monitoring. It lives out in the wild, in the messy, unpredictable swirl of production. It watches actual traffic, actual data, actual human beings clicking buttons. When a monitor fires, the damage isn’t theoretical anymore. It’s already happening. The alert might show up in seconds, sure—but during those seconds, you might have dropped ten thousand requests, corrupted a batch of records, or quietly lost a chunk of revenue. The monitor tells you the building is burning. It doesn’t stop the arsonist.
That’s not a dig at monitoring. We’d be dead without it. But monitoring is, by its very nature, reactive. Testing is preventive. Calling them interchangeable isn’t a minor oversight; it’s a category mistake. You wouldn’t say, “We don’t need a firewall because we have an incident response team.” Same logic.
The Dashboard’s Blind Spots
Monitors run on metrics, logs, and traces. Each of those has holes you could drive a truck through.
Metrics love to average things out. A p99 latency number can look perfectly healthy while a tiny sliver of requests are failing outright. Your CPU graph might be a calm blue line, even as a single thread sits there deadlocked, doing absolutely nothing. The aggregate view is great at hiding the weird little anomalies that actually matter.
Logs are only as smart as the person who decided what to log. If an engineer never instrumented a particular code path—maybe because it seemed too obvious to fail—then those failures produce exactly zero log lines. The system crashes in total silence. Your monitor won’t even twitch.
Traces are supposed to give you that beautiful end-to-end story, but only if every service in the chain plays nice and propagates context without dropping it. A missing header, a misconfigured sampler, some library that doesn’t support your tracing protocol—suddenly you’ve got gaps. And the worst part? Those gaps are invisible from your dashboard. You don’t know what you’re not seeing.
In every case, the monitor shows you what you told it to look for. It can’t show you what you never thought to instrument. Testing—especially the weird stuff like property-based testing or chaos experiments—pokes the system in ways you didn’t anticipate. It hunts for the unknown unknowns that monitoring, by design, will miss.
The Coverage Fantasy
I’ve heard this pitch a hundred times: “We monitor all our critical paths. If anything breaks, we’ll know in minutes.” The quiet assumption underneath is that critical paths are the only places where things can go spectacularly wrong.
That assumption fails constantly, and often in embarrassing ways. A so-called non-critical path suddenly becomes very critical when it interacts with a critical path in some unexpected fashion. A logging library that blocks on disk I/O can freeze an entire request pipeline. A background job that leaks memory at a trickle can bring a server to its knees after six hours of totally normal looking metrics. You stare at the dashboard and see latency spiking on the critical path, but the actual root cause is sitting in a component nobody bothered to test properly.
Testing makes you stare directly at those interactions. A decent integration test prods the boundaries between components. A load test would’ve surfaced that memory leak before it ever hit production. Monitors can tell you the system is degrading. They almost never hand you the “why.”
When Monitoring Becomes the Crutch
Teams that chronically underinvest in testing start to develop a certain rhythm. They ship a change. They huddle around the dashboards with their pulse elevated. When something spikes, they roll back. The release process turns into a coin toss, with monitoring as the safety net they pray they won’t need.
This rhythm has real, compounding costs. Rollbacks chew up engineering time and burn user trust. The feedback loop is painfully slow—you find out about problems only after real people have already suffered through them. And the pattern poisons any appetite for serious refactoring, because nobody on the team actually trusts that the system will hold together under change.
Testing gives you that trust. A solid test suite doesn’t promise zero bugs—nothing does—but it does guarantee that the behaviors you know about stay intact. When a test fails, you know exactly what broke and precisely where. When a monitor screams, you’re starting a frantic investigation from zero, under pressure, while your users are already feeling the pain.
These two feedback loops operate at totally different speeds and with wildly different precision. Choosing the slow, fuzzy loop because it looks easier than building the fast, sharp one? That’s a trade-off that gets more expensive every single sprint.
The Accounting Trick
Writing tests takes time. Running them burns compute. Maintaining them is ongoing work. Monitoring also demands time, compute, and maintenance. But monitoring often looks cheaper on paper because its costs are spread out differently and because the pain of skipping it is more immediately visible.
That’s an accounting illusion, plain and simple. The real price tag of a production incident includes more than just the engineering hours to patch it. There’s lost revenue, a dinged-up reputation, and the opportunity cost of all the real work that didn’t get done. These costs are squishier to measure than a Jira ticket for writing tests, so we tend to discount them or pretend they don’t exist.
When you actually compare the cost of a test suite against the cost of the incidents it prevents, the test suite usually comes out ahead by a wide margin. But that calculation forces you to imagine counterfactuals: the incidents that never happened because a test snagged the bug early. Humans are notoriously terrible at valuing things that didn’t occur. We’re wired for stories, not statistical absences.
Monitoring, on the other hand, gives you visible, countable events. Every alert has a story. Every incident gets a postmortem. The value feels concrete, even when the incident itself is just a monument to a failure of prevention.
The Real Relationship
None of this means you should pick testing and throw monitoring in the trash. That’s a false choice. The two feed each other. Testing lowers the odds of failure. Monitoring shrinks the blast radius when a failure inevitably slips through. A mature engineering practice does both, and it does them well.
The trouble starts when monitoring gets used as an excuse to skip testing. That’s like arguing that because you’ve got great health insurance, you don’t need to exercise or eat a vegetable ever again. The insurance helps after the heart attack. It does absolutely nothing to keep your arteries clean.
These practices also talk to each other. A production incident often shines a flashlight on a hole in the test suite. A test that keeps catching regressions is a big neon sign that this part of the system deserves tighter monitoring. Done right, the two create a loop that makes the whole system tougher over time.
The Failures Monitoring Misses
Let’s get specific about a few failure modes that sail right past your average dashboard.
Correctness errors. A function that quietly computes the wrong number for certain inputs might not trip any metric alarm. The system stays up. Latency looks fine. Error rates are flat. But the output is wrong. Monitoring won’t catch this unless you’ve baked explicit correctness checks into your metrics—which is really just a form of testing that happens to run in production.
Security vulnerabilities. An authentication bypass that generates no errors and no weird traffic patterns can live in production indefinitely if you’re only watching metrics. The monitor sees valid requests getting served. It has no idea those requests should have been blocked.
Slow-motion data corruption. A bug that flips a single bit in a fraction of a percent of database writes likely won’t cross any threshold. The corruption piles up silently, and by the time it’s obvious in the metrics, the damage is massive and possibly irreversible.
Bizarre business-logic edge cases. A discount calculation that only breaks on February 29th. A shipping estimator that loses its mind for addresses in one specific postal code. These won’t show up in aggregate metrics. They hit a tiny number of users, but for those unlucky few, the experience is memorably awful.
Testing can tackle each of these head-on. Unit tests nail correctness. Security tests poke at vulnerabilities. Property-based tests hunt for those edge cases like a bloodhound. Monitoring can backstop these efforts, but it can’t replace them.
The Visibility Problem
There’s a sociological layer here that’s hard to ignore. Monitoring is loud. Dashboards glow on big screens in the office. Alerts blast into Slack channels. The monitoring stack itself is a thing you can point at and say, “Look what I built.”
Testing is quiet to the point of being invisible. A green test suite gets no applause. The grind of writing tests is often completely hidden from anyone outside the engineering team. A manager who never reads code might go years without seeing a test file.
This visibility gap creates a warped incentive structure. Monitoring investments generate observable output. Testing investments generate the absence of observable problems. It takes real organizational maturity to value the second as much as the first.
The strongest engineering leaders I’ve seen understand this dynamic and actively fight against it. They ask about test coverage in the same sentence as monitoring coverage. They treat a skipped test with the same weight as a missing alert. They know that the quiet work is what keeps the loud stuff from happening.
A Framework That Actually Works
Here’s a simple way to think about it. For any given failure mode, ask two questions:
- Can a test catch this before it reaches production?
- Can a monitor catch this after it reaches production?
If the answer to the first question is yes, write the damn test. The monitor is a backup, not your primary defense.
If the answer to the first question is no—maybe the failure depends on production data or traffic patterns you can’t replicate in a staging environment—then monitoring is your only real tool. Pour resources into it and make it sharp.
If the answer to both is no, you’re staring at a true blind spot. That failure will happen, and you won’t know about it until a confused user files a support ticket. These are the scariest failures, and they deserve your full, undivided attention.
This framework clears up the confusion instantly. Testing and monitoring answer different questions. When you treat monitoring as a stand-in for testing, you’re essentially answering “no” to that first question without even checking. That’s not engineering. That’s gambling.
The Evidence Is Everywhere
I’ve read more outage postmortems than I care to count, and they all follow the same script. The incident starts with a change: a deployment, a config tweak, a dependency bump. The change introduces a failure mode that had zero test coverage. The failure surfaces in production—sometimes right away, sometimes hours later after a slow burn. Monitoring catches the symptom, but by then the damage is done. And the postmortem’s action items? They always, always include adding tests for that specific failure mode.
Notice the order. The tests come after the incident, as a remediation. They should have been there before. The monitoring did its job—it detected the mess—but it didn’t prevent it. The organization learned something, sure, but the tuition for that lesson was paid in downtime and user pain.
This pattern repeats because the cost of writing tests is felt right now, while the cost of skipping them is deferred and probabilistic. Humans, and the orgs we build, are lousy at making decisions under these conditions. We discount future pain. We overvalue visible activity. We whisper to ourselves that this time will be different.
The engineering answer is to build systems that override these cognitive biases. A CI pipeline that hard-blocks deployment on test failures is a perfect example. It enforces testing not because everyone is disciplined enough to remember, but because the system makes it physically impossible to skip.
Wrapping Up
Monitoring is not a cheaper version of testing. It’s a different instrument, built for a different job, operating in a different phase of the software lifecycle. Confusing the two gives you systems that break in thoroughly predictable ways, with consequences you could have dodged.
The relationship isn’t either-or. It’s both, and it’s deliberate. Test aggressively to block the failures you can see coming. Monitor aggressively to catch the ones you can’t, and to measure the damage from the ones that sneak through. Each practice sharpens the other.
Next time someone waves their hand and says monitoring means you can go easy on testing, ask them one question: “What actually happens when the monitor finds something?” The honest answer is almost always that users have already been hit. Testing is the discipline that keeps the monitor boring. And a boring monitor is the single best signal of a healthy system.
Frequently Asked Questions
Can’t we lean on canary deployments and monitoring instead of heavy pre-production testing?
Canary deployments shrink the blast radius, but they don’t stop the explosion. If a bug torches 1% of your users in the canary, those users still had a rotten time. Monitoring tells you the canary is on fire, but the fire already happened. Pre-production testing catches the bug before a single real human sees it. Canaries and monitoring are excellent safety nets, but they work best when the net is almost never used because testing already stripped out most of the risk.
Which types of testing get dropped first when teams over-rely on monitoring?
Integration tests and property-based tests are usually the first to go. Unit tests are relatively cheap and easy, so they hang on. But integration tests—the ones that verify components actually work together—are harder to set up and slower to run. Property-based tests force you to think about invariants and generate weird inputs, which takes real effort. When monitoring is treated like a substitute, these higher-effort, higher-value tests get abandoned. The result is a system where individual pieces look fine in isolation, but their interactions produce ugly surprises in production.
How do you convince a monitoring-obsessed team to invest in testing?
Start by quietly tracking the cost of production incidents that a test could have caught. For each incident, ask: “Would a test have caught this?” If the answer is yes, log it. Over time, you build a pile of evidence that connects skipped tests to actual pain. Present that evidence not as a blame game, but as a straightforward investment case: the hours lost to incidents dwarf the hours it would have taken to write the tests. Also, make testing less painful. If the test suite is slow, flaky, or a nightmare to run, fix that first. People avoid testing when it hurts. Take away the pain, and the behavior often shifts without a single argument.