Monitoring and Observability: What Production Software Actually Needs

Monitoring tells you something's wrong; observability tells you why

These terms get used interchangeably, and the distinction actually matters for monitoring and observability software done right. Monitoring is watching known signals against known thresholds — CPU usage, error rate, response time — and alerting when they cross a line. It answers whether something is wrong. Observability is the broader capability to ask new questions about a system's internal state without having predicted the question in advance — to look at a specific failing request and trace exactly what it touched, what it returned, and where it diverged from a healthy one. Monitoring catches the failures you anticipated. Observability is what lets you actually debug the ones you didn't.

The three pillars, and why you need all of them

Logs, metrics, and traces each answer a different question, and a production system missing any one of them has a real blind spot. Metrics tell you something changed — error rate jumped at a specific time. Logs tell you what happened around that time in a given service. Traces tell you the full path a single request took across every service it touched, which is the only way to debug a slow or failing request in a system with more than one moving part. We've inherited plenty of codebases with solid logging and zero tracing, and every performance investigation in those systems turned into hours of guesswork instead of following one clear trail.

This gap tends to show up specifically once a system stops being a single application and becomes a handful of services talking to each other — a request that's slow somewhere across four service calls is nearly impossible to debug from logs alone, because you're manually correlating timestamps across systems that don't share a common request identifier. Adding a trace ID that propagates through every service from the moment a request enters the system is a small change that pays for itself the first time it turns a multi-hour investigation into a five-minute lookup.

Alert fatigue is a design failure, not a tooling problem

An alerting setup that pages someone for every anomaly, regardless of severity, trains the team to ignore alerts — and once that happens, the real incident gets the same shrug as the false alarm that preceded it. This isn't a tooling limitation; it's a design choice made poorly. We tune alerts to page only on things that require immediate human action, route lower-severity signals to a dashboard or a non-urgent channel, and treat every page that turns out to be a non-issue as a bug in the alerting configuration that needs fixing, not a Tuesday. A team that trusts its alerts responds faster to the ones that matter.

An alert nobody trusts is worse than no alert at all — it costs you the same attention with none of the signal.

What to actually instrument, in priority order

We've also seen the opposite failure — teams that instrument everything indiscriminately, generating so much telemetry that the signal is buried in noise and the observability bill itself becomes a line item worth optimizing. More instrumentation isn't automatically better instrumentation. The question we ask before adding any new metric or trace is whether it would actually change what someone does during an incident. If the answer is no, it's not worth the storage cost or the extra noise in the dashboard.

We start instrumentation with the paths that directly affect revenue or core user experience — checkout, authentication, the primary action your product exists to perform — before spending time on secondary features. Within that, we prioritize error rates and latency percentiles, specifically the tail end and not just averages, because averages hide the tail-end experiences that are usually the ones users complain about, alongside business-level metrics that map to what the product is actually for, like completed signups or successful transactions, not just infrastructure health. A system can look perfectly healthy on CPU and memory graphs while the core user flow is silently broken.

Dashboards nobody looks at are worse than no dashboard

We've seen impressive-looking dashboards built early in a project that nobody consults during an actual incident, because the person debugging doesn't know they exist, doesn't trust the data, or the dashboard answers a question nobody actually asks in practice. A dashboard is only valuable if it's part of the team's actual incident response habit — which means building it around the questions engineers ask when something's broken, not around the metrics that were easiest to collect. We test this directly: during a postmortem, if the dashboard didn't help answer what happened, it gets rebuilt or removed.

Observability is a cost center until the day it isn't

It's easy to deprioritize this work because a healthy system doesn't seem to need it — until the day it very obviously does, and the gap in your instrumentation turns a fifteen-minute fix into a four-hour investigation with a customer-facing outage running the whole time. We build monitoring and observability software into every project from the start, proportional to its stage, because the alternative isn't saving effort — it's deferring the cost to the exact moment you can least afford to pay it.

If you're scoping something like this, see our cloud & DevOps services.