News

​Sai Vishnu Bhyravajosyula, Principal Software Engineer at Atlassian.

gettyThe incidents that cause the most lasting damage are the quiet ones, where the system continues serving traffic without interruption, while customers receive incorrect results.

I’ve sat through many incident-review meetings in my career. The dramatic ones, where everything goes down at once, are painful but at least they’re honest. Something broke, and you go dig it out.

During the call that changed how I design systems, every metric looked fine. Replication was verified, failover had been tested a few days earlier, and latency was sitting right where it always sat. Then, a few of our bigger customers started getting stale data without us receiving any reported errors or alerts.

A customer caught it before we did. That one bothered me for a long time, because we were proud of our observability. It turned out to be only capable of saying whether the system was up and running, but it had nothing to say about whether the system was correct.

What A Dashboard Actually MeasuresIn my experience, most observability gets built reactively. You ship something and add uptime monitoring first. Latency tracking shows up later, usually after someone gets alerted about slowness. The dashboard grows panel by panel, and after a while it starts to feel like real visibility.

But even when it tells you the pipes are flowing, that doesn’t mean the water is clean. For a long time, that was a fine trade-off. Back when systems were mostly single-region, infrastructure health and data correctness moved together. If the system was up, the data was probably right, too. That assumption no longer holds once a system becomes distributed.

Picture two users in different regions writing to the same record within milliseconds of each other. Both writers succeed. Each region now believes it holds the latest version, but they can’t both be right.

And that’s the clean version. Schema migrations roll out unevenly across regions. Retry storms during a degraded state don’t heal divergence; they widen it. A failover that passed every staging test we had once kicked off a downstream cascade no one had thought to model.​

Dashboards look healthy, but the system isn’t. That gap, between a system that works and one that has learned to look like it works, is where the expensive failures live.​

What Production Taught MeA while later, I worked on a platform that moved data between regions in bulk, tens of millions of records a week, with the constraint that certain records legally could not cross some of those boundaries.

We were deep in the design review when one of the engineers, someone who mostly stayed quiet in these meetings, asked a question that stopped the room: If our data started diverging between regions right now, how long before we’d actually know?

No one had an answer. We knew our monitoring would catch an infrastructure failure within a couple of minutes. A correctness failure we’d find out about whenever a customer noticed their numbers didn’t add up and bothered to escalate.

That question reshaped the whole design. We added hash-based verification between replicas and ran it continuously. We built checkpoint capture into a dedicated table tracking verification statuses at each stage, so we always knew exactly where data stood between source and destination.

Recurring ad hoc verification workflows also spot-checked records across both ends throughout the migration window, not just at cutover. When checks flagged inconsistencies, a reconciliation workflow replayed and corrected affected records without a full re-migration.

For the most critical writes, we used a dual-write pattern during transition, writing to both sides simultaneously and comparing outcomes before committing.

Finally, we wrote correctness service-level agreements (SLAs) and put them on the same dashboard as the infrastructure SLAs, right next to uptime, and that placement did more to change the team’s habits than the checks themselves.

In its first week, the verification flagged divergences we would never have seen otherwise. It was the first time we were watching whether the system was correct rather than just whether it was alive.

What Leaders Can Actually DoIf I could hand you just four things, it’d be these:

Measure your time-to-detect on correctness. Put it to your team directly: If our data started diverging between regions right now, how long before we’d know?

Don’t accept a reassurance about monitoring coverage. Push for a number. Most teams have never been asked this out loud, and the silence you get back tells you where your exposure sits.

Reframe correctness as customer trust. This is the reframe that decides whether correctness gets funded. In a reliability backlog, it loses to feature work almost every time.

Call it a trust problem, however, and it moves. Point out that silent degradation reaches customers as churn and support tickets weeks before anything shows up on an error graph.

Pay for correctness observability like it’s infrastructure. Continuous replica comparison and hash checks, mostly. Checksums too, though those earned their keep less than we expected.

If your data has business consequences, this is table stakes. Almost everyone has strong infrastructure observability and almost nothing on the correctness side, and that gap is rarely an oversight.

Design for divergence before it designs your postmortem. Most correctness incidents I’ve seen rhymed. The failure was findable in advance, but the team had just never modeled it, so a customer noticed first.

Work out what divergence would look like, how you’d catch it and what you’d do while you’re still in the architecture review. That conversation is cheaper at 2 p.m. than at 3 a.m. with customers already down.​

ConclusionThe outage you can see is rarely the one that does lasting damage. The real cost comes from the system that stays green while serving wrong answers, week after week, and your monitoring never points it out.

I spent years shipping systems that passed every check we had. Some were fine, some were faking it, and I couldn’t tell which until a customer told me. The job now is catching that difference first.​​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?