The Shape Under the Average

Most dashboards report the number. The number gets rolled up, presented in a slide, and absorbed into the operating narrative. Within a quarter, it’s authoritative. Within a year, nobody remembers where it came from.

The number on the slide is rarely wrong in the mechanical sense. It’s usually accurate to the data path that produced it. The problem is that the data path makes assumptions about what to include, what to exclude, what to combine, and what to treat as the same thing.

My work is finding the shape underneath the aggregate.

A few patterns I keep returning to.

Aggregates hide structure. A platform-wide metric that sits in the middle of its possible range usually isn’t describing a population that behaves in the middle. It’s describing a population that doesn’t exist as a coherent group. The average is the math result of combining two or three distinct populations into one number. The strategic move isn’t to improve the average. It’s to ask which sub-population the strategy is actually targeting.

In one analysis, a platform-wide adoption metric appeared to show a population sitting near the middle. The distribution showed something different: most accounts were clustered at the extremes. The conversion question changed once the shape was visible. The middle wasn’t the target. One of the poles was.

Segmentation comes before interpretation. Two different transaction types, account segments, or platform versions can be aggregated into one metric. The aggregate hides the divergence. When the two populations behave differently, the aggregate is mathematically correct and strategically misleading.

The first analysis is almost never “what does this metric say.” It’s “what populations is this metric combining, and do they belong together.”

The data path is part of the metric. Numbers that have been authoritative for months can be wrong, not because of measurement error but because of definition drift. A pipeline that filtered out a category three years ago for a reason that no longer applies will still filter it out today. The number stays consistent. The consistency is itself the problem.

When an institutional number doesn’t hold up under a direct query against canonical sources, the answer is rarely that the data is wrong. It’s that someone, somewhere, made a definition choice that nobody has revisited.

Rising numbers are not always good news. Vanity metrics can grow for the wrong reasons. Traffic spikes without corresponding conversion movement are usually bot inflation or off-channel referral noise. Adoption metrics that grow during a vendor’s promotional period are pricing-driven, not value-driven. The metric going up is the wrong thing to celebrate. The pattern in the underlying behavior is the signal.

The interrogation is the same in either direction. Whether the number is going up or down, the question is whether the movement reflects what you think it reflects.

Dead findings make live findings credible. Two analyses I produced last year got invalidated on the record before reaching leadership. One was a structural gap that turned out to be inflated by category mixing. The other was a friction signal that turned out to be a scoring artifact reading normal operational boundaries as user friction.

Both got documented and discarded.

The discipline of publishing dead findings alongside live ones is what makes the live ones credible. When you publish only what survived, you’re asking the reader to trust your filter. When you publish what didn’t survive, you’re showing them how the filter works.

This is the part of the practice that I think matters most. It’s also the part most institutions reward least. Killing your own work is unrewarded. Defending it well past its useful life is, in the short term, often more career-friendly. The discipline of doing the harder thing is the thing that compounds.

Across all five patterns, the move is the same: never trust an aggregate until you’ve understood the shape underneath it.

The number is usually fine.

The interpretation, almost never.