You glance up at the dashboard. Everything is green. Uptime 99.98%. Latency hovering around 110ms. Error rate a flat 0.01%. Somewhere a junior engineer exhales, cracks a joke about how easy the week has been, and shuts the laptop early. Nora Ishikawa does not exhale. Nora Ishikawa has learned, usually the hard way, that a spotless monitoring board isn’t evidence of a healthy system. It’s evidence of a system nobody has bothered to interrogate properly.
Monitoring tells you what the system is doing right this second. Testing tells you what the system doesn’t yet know how to survive. These aren’t adjacent disciplines you can swap out depending on how busy the sprint is. They’re fundamentally different stances toward risk, and mixing them up is how you ship brittle code while looking awfully pleased with yourself.
The False Comfort of Failures You’ve Already Met
Monitoring runs on thresholds. CPU over 85% for five minutes? Page someone. Disk queue length climbing? Fire an alert. These are known failure modes. Someone, at some point, guessed this metric could degrade in this particular way and wired up a check. That’s useful. It’s genuinely necessary. But it’s also the shallowest safety net you can get away with. You’re protecting yourself against failures you already imagined. The failures that actually trash systems, ruin weekends, and quietly eat away at user trust are the ones nobody saw coming. Those don’t trigger alerts because nobody set a threshold for a condition nobody predicted.
Testing, when you do it right, is an adversarial act. A solid integration test doesn’t politely ask, “Does this endpoint return 200?” It asks, “What happens when the downstream payment service sends back a 200 with a body that’s complete garbage?” Or, “What if the message broker acknowledges the publish but quietly drops the message?” These aren’t conditions you’ll find on a standard dashboard. They show up only when you deliberately stir up chaos, violate your own assumptions, and see what crumbles before your users get a chance to.
The Observability Trap
The industry spent half a decade preaching observability: structured logs, distributed traces, high-cardinality metrics. The promise is genuinely seductive. Instrument everything, and you can ask arbitrary questions of your system in real time. Debug in production. And to be clear, that capability is real and powerful. But it has a shadow side: it encourages a reactive posture so comfortable that proactive testing quietly withers away.
Here’s a pattern Nora has watched play out more times than she cares to count. A team deploys a service. They wire it into their tracing pipeline. They set up dashboards for p99 latency, error ratios, and throughput. Then they stop. The service runs. The graphs look reasonable. Nobody writes a test that injects a three-second GC pause into the runtime because “we’d see it in the traces if it happened.” But by the time it happens in production, the trace only confirms the damage. The damage is already done. The trace is an autopsy, not a vaccine.
When “Fast Detection” Morphs into a License to Skip Work
There’s a breed of engineer—often talented, often well-intentioned—who argues that with fast enough detection and automated rollback, you can ship untested code safely. The logic goes: if we can spot a regression in under two minutes and revert in under one, isn’t that roughly the same as preventing it? No. It is not. It’s roughly the same as accepting a three-minute user-facing incident as the cost of doing business, multiplied by however many times your fast detection actually works. And that number is never 100%.
Detection latency is never zero. Rollback latency is never zero. The blast radius during those minutes can include corrupted data, poisoned caches, cascading failures in dependent services, and user sessions torn down mid-transaction. A test that runs in CI for 90 seconds prevents all of that. A dashboard that fires an alert 90 seconds after deployment does not. One stops the bullet. The other just measures the entry wound.
The Quiet Rot of Unverified Invariants
Every system runs on invariants: assumptions that must hold true for anything to be correct. “User IDs are never negative.” “Inventory counts never dip below zero.” “A confirmed order always has exactly one payment record.” Monitoring can check some of these, but only if you explicitly codify them as metrics or log-based queries. Most teams don’t. They monitor the symptoms of invariant violations—spiking errors, weird latency—but not the invariants themselves. And invariants don’t always break loudly. Sometimes they break quietly, corrupting state that surfaces three weeks later in a billing reconciliation nightmare that nobody has the stomach for.
Testing, particularly property-based testing and data-integrity checks that run on deploy, can verify invariants directly. You can write a test that generates thousands of randomized order states and asserts that the “confirmed” invariant holds across every single one. You cannot do that with a dashboard. A dashboard shows you a time series of a metric you chose. A test explores the state space of your system’s actual logic. These are not the same activity, and no number of Grafana panels will close that gap.
The Economics Are Backward
A common objection: “We don’t have time to write comprehensive tests. We invest in monitoring because it’s cheaper.” This is flat wrong in any system that survives longer than a quarter. The cost of a production incident isn’t just the engineering hours to patch it. It’s the lost trust of users, the degraded brand perception, the support tickets, the internal escalations, the context-switching tax on every engineer who gets paged, and the opportunity cost of whatever those engineers would have built instead. A single severe incident can erase months of testing investment without breaking a sweat.
More subtly, monitoring without testing creates a false economy of speed. You ship faster initially because you skip the test-writing step. But you also ship regressions faster. Over time, the system becomes a minefield. Engineers grow afraid to refactor. Deployment frequency drops because every deploy feels like a bet you might lose. The very velocity that skipping tests was supposed to protect evaporates, replaced by a culture of low-grade fear and an ever-thickening stack of runbooks describing how to manually verify things that should have been automated from the start.
The Refactoring Ceiling
Monitoring-dependent systems hit a refactoring ceiling. When you lack a trustworthy test suite, any structural change—upgrading a framework, splitting a monolith, changing a database schema—carries an unquantifiable risk. The monitoring says the system is healthy now, but it cannot tell you whether the new code path preserves the behavior of the old one under edge conditions. So you either avoid the refactor, accumulating technical debt, or you do it and hope, which is just gambling with a professional veneer.
Testing removes that ceiling. A system with fast, deterministic tests can be restructured aggressively. You can extract a service, change a data model, swap a library, and have high confidence that if the tests pass, the externally observable behavior is preserved. Monitoring tells you the system is alive. Testing tells you the system is correct. Alive and correct are not the same thing, and confusing them will eventually cost you a night you’ll remember for all the wrong reasons.
Where Monitoring Excels (And Why That Still Isn’t Enough)
None of this is an argument against monitoring. Monitoring is indispensable for capacity planning, for detecting slow-burn resource leaks, for understanding user behavior patterns, and for diagnosing incidents that slip past your tests—because they will. Tests are not perfect. They cannot simulate every production condition. Network partitions, disk failures, DNS resolution delays, AWS us-east-1 having another bad day: these are the domain of monitoring and observability. The mistake is treating monitoring as a substitute for testing rather than a complementary, second line of defense.
The correct mental model is layered. The innermost layer is static analysis and type checking, catching errors before the code even runs. The next layer is unit and integration tests, verifying logical correctness. Then contract tests, ensuring service boundaries hold. Then deployment-time smoke tests and canary analysis. Only then, at the outermost layer, does monitoring sit, watching for the unknown unknowns that slipped through every previous layer. If your outermost layer is the only layer, you haven’t built a safety system. You’ve built an alarm clock for disasters.
A Practical Litmus Test
Nora proposes a simple exercise. Find your most critical user journey—placing an order, uploading a file, starting a stream. Now ask: if the database for this journey started returning every third query with a 200-millisecond delay but no errors, would your monitoring detect it? Probably yes, as a latency bump. Now ask: if the database started returning stale data—data committed by another transaction but not yet visible to this one—would your monitoring detect it? Almost certainly not. The errors would be silent, showing up only as confused users and support tickets days later. A well-designed test with isolation-level assertions would catch it in a pipeline. Your dashboard would not.
This is the gap. It’s not small. It’s the gap between measuring the system’s vital signs and verifying its actual behavior. Monitoring is a stethoscope. Testing is a blood test. You can hear a heartbeat and still be dying.
FAQ
Can’t synthetic monitoring replace some types of testing?
Synthetic monitoring—running scripted user journeys against production at regular intervals—is valuable, but it is not a test suite. It runs a fixed set of happy-path scenarios, usually from outside the system, and it runs them infrequently relative to a CI pipeline. It cannot explore edge cases, cannot inject faults into internal components, and cannot run before code reaches production. It’s a sanity check, not a verification mechanism. Treating it as a test replacement is like checking your car’s oil once a month and calling it a full engine diagnostic.
How do you convince a team that prioritizes monitoring to invest more in testing?
Don’t argue in abstractions. Collect data from the last six months of production incidents. For each incident, ask two questions: “Could a test have caught this before deploy?” and “Did our monitoring catch this before users were affected?” Present the ratio. In most organizations, the answer to the first question is “yes” more often than anyone wants to admit, and the answer to the second reveals that monitoring often detects issues only after user impact. Concrete incident retrospectives, framed without blame, are far more persuasive than theoretical debates about testing philosophy.
Where should a team start if they have good monitoring but minimal testing?
Start with the scariest thing. Identify the two or three components where a failure would cause the most damage—revenue loss, data corruption, regulatory exposure. Write tests for those first, focusing on the known incidents from your retrospective. Then add tests for the invariants you’ve never seen break but that would be catastrophic if they did. Do not try to backfill a comprehensive test suite overnight; that’s a recipe for burnout and abandoned efforts. Build the habit incrementally, proving the value with each prevented incident, until testing becomes as non-negotiable as the dashboard screens on the wall.
