The first time I watched an AI-generated test pass for the wrong reasons, it sat in a pull request for a payment-settlement module. The test had the right shape: it initialized a client, called a function, asserted a status code. The name even looked thoughtful—test_settlement_retry_on_failure. But the assertion checked for Continue Reading
The False Comfort of Dashboards: Why Monitoring Never Replaces Testing
I sat in a post-mortem once where a team lead gestured at a Grafana dashboard and said, with zero irony, “But all the metrics were green.” The system had been down for forty minutes. Customers noticed. The dashboards didn’t. That sentence stuck with me because it captures a quiet, pervasive Continue Reading
Green Lights Don’t Mean Go: Why Monitoring Alone Is a Reliability Trap
There’s a silence that settles over the NOC at 3 a.m. that I’ve learned to distrust. Every dashboard glows green. CPU loafing at 40 percent. Memory a flat line. Error rate sitting at a smug zero. I’m there with a cold cup of coffee, and I know that silence isn’t Continue Reading
The False Comfort of Green Lights: Why Monitoring Is Not a Substitute for Testing
There’s a quiet, persistent belief that spreads through engineering teams like a slow leak. It goes something like this: if the dashboards are green, if the alerts are silent, if the metrics sit inside their thresholds, then the system is healthy. This belief isn’t just wrong—it’s dangerous. Monitoring tells you Continue Reading
Your Alerting Dashboard Is Feeding You a Comfortable Lie
Another incident report landed on my desk. Same story. Monitoring flagged a spike in p99 latency. The on-call engineer got the page, then burned forty minutes restarting services, cycling connection pools, and finally rolling back a release that had been running fine for six hours. The rollback “fixed” it. The Continue Reading
Your Dashboard Is Full of Green Lights and Half of Them Are Lying
You glance up at the dashboard. Everything is green. Uptime 99.98%. Latency hovering around 110ms. Error rate a flat 0.01%. Somewhere a junior engineer exhales, cracks a joke about how easy the week has been, and shuts the laptop early. Nora Ishikawa does not exhale. Nora Ishikawa has learned, usually Continue Reading
Why Your Monitoring Dashboard Won’t Save You From a Broken Build
There’s this quiet, stubborn idea that floats around engineering orgs. It goes like this: once you’ve got enough monitors wired up, you can finally relax on the testing front. The dashboards will glow red when something’s wrong. PagerDuty will scream. Someone will fix it. And all those slow, fiddly integration Continue Reading
Why Edge Cases Are Not Edge Cases in Production
Engineers love the phrase edge case. It slips out during design reviews, code inspections, sprint planning. Someone flags a weird input, a timing hiccup, a user behavior that doesn’t map to the happy path, and the label lands: edge case. Then, almost as a reflex, the quiet corollary—we don’t need Continue Reading
Why Edge Cases Are Not Edge Cases in Production
The Lie We Tell Ourselves During Development Every engineer has muttered it: “That’s an edge case.” The phrase closes tickets, defers fixes, and blesses designs that only survive under perfect conditions. In the sterile bubble of a dev branch—clean test data, a single user pinging the endpoint—these rare scenarios feel Continue Reading
The Edge Case Is a Lie We Tell Ourselves at 3 a.m.
A system folds at 3 a.m. and the postmortem reaches for that comfortable phrase: “edge case.” I’ve heard it in too many war rooms. It’s a quiet surrender—a way of saying the failure was improbable, exotic, and somehow forgivable. But production doesn’t forgive. It takes those supposed rarities and turns Continue Reading