I once convinced a small family-owned medical supply business to spend $4,420 on a premium multivariate testing suite, despite the fact that their monthly unique visitors barely cracked the 800-person mark.

Although the software was marketed as a “growth engine” for any scale, I was essentially asking a calculator to perform a séance. I stood in their wood-paneled office, listening to the hum of a desktop fan that sounded suspiciously like the opening chords of “99 Red Balloons”-a song that has been looping in the back of my skull for three days-and I told them that data would set them free. I was wrong, and I was wrong in that specific, expensive way that only a consultant desperate to look scientific can be.

The reality was that we were trying to measure the “lift” of a button color when the total number of conversions for the month could be counted on the fingers of a very talented pianist. We spent six weeks “optimizing” a checkout flow that only fourteen people saw. It was a mathematical farce, a performance of rigor performed for an audience of none, yet I defended it because the dashboard looked authoritative.

The Wednesday Morning Hallucination

Wednesday mornings usually bring this realization to a head during the growth stand-up. You sit there, caffeine-jittery and vaguely annoyed by the persistent melody of that German synth-pop track, watching a screen share of a testing tool. The dashboard shows Variant B at a 4.2 percent conversion rate against a 3.5 percent for the Control.

The little bar is highlighted in a cheerful, confident green. Someone in the back of the room, perhaps the only one still tethered to the physical world, asks the uncomfortable question: “How many actual sales is that?” There is a pause, a series of clicks that echo like a tintinnabulation in the silence, and the truth is revealed.

Nineteen conversions versus sixteen. The team nods, satisfied. They ship Variant B, close the Jira ticket, and walk away feeling more data-driven than they did ten minutes prior.

It is the noise of the universe masquerading as a signal. When you are running experiments on nine hundred visitors a month, you aren’t testing a hypothesis; you are reading tea leaves with a very expensive digital magnifying glass. The apparatus of enterprise-level rigor arrives in these smaller organizations long before the traffic that would make such rigor actually rigorous.

This happens because the software is cheap and accessible, but the volume-the actual human attention required to make a p-value meaningful-is the most expensive commodity on earth.

The Clinical Reality of Small Samples

In my work as an advocate for families navigating elder care, I see a similar obsession with “quality metrics” that often obscures the human reality. How this actually works in a clinical setting is a process called “small-n” analysis, where you cannot rely on large-scale statistics because you are only looking at twelve residents in a memory care wing.

In that environment, if three people fall in July and five people fall in August, you do not declare a “66 percent increase in safety risk” and overhaul the entire flooring system. You look at the individual stories. You realize Mrs. Higgins got a new pair of slippers that are too slippery, and Mr. Miller had a change in his blood pressure medication.

You solve the problem with observation and intuition, not by worshipping a percentage sign. We have lost this ability in the digital space. We have been told that “measurement always beats intuition,” but that is a half-truth that ignores the threshold of validity.

Below that threshold, measurement is just intuition wearing a costume. It is a way to outsource the terrifying responsibility of making a choice. If you choose to change the headline because you think it’s better, and it fails, it’s your fault. If you change it because the dashboard gave you a green checkmark based on nineteen people, it’s the data’s fault.

We are buying a $500-a-month insurance policy against being blamed for our own creative decisions. This creates a stultifying environment where the six weeks spent running a statistically insignificant test are six weeks not spent doing the actual work.

Math is a Jealous God

There is a certain bathos in watching a room full of smart people debate a three-person delta. It’s a collective hallucination. We want to believe we are scientists because science implies a level of control over the chaotic marketplace. If we can measure it, we can master it.

But math is a jealous god; it doesn’t care about your desire for certainty. If you don’t have the numbers, the math simply isn’t there, no matter how many times you refresh the page. Organizations adopt these instruments because they confer legitimacy.

If you want to get a budget approved for a redesign, it is much easier to point to a “validated experiment” than to say, “The current site looks like it was designed by a committee of people who hate joy.” The vocabulary of testing-statistical significance, confidence intervals, multivariate clusters-is a shield. It makes it difficult for anyone to question the direction without sounding like a Luddite.

When I look at the work done by a high-level shop like

Coherent Agency, I notice a distinct lack of this performative “growth hacking” for the sake of it. They understand that a pixel-perfect Webflow build and a robust CMS architecture are worth more than a hundred failed A/B tests on a broken foundation.

You cannot optimize your way out of a fundamentally flawed brand or a confusing user experience. You can’t put a spoiler on a car with no engine and expect it to win a race just because you measured the wind resistance on the spoiler.

The Psychology of Rounding Errors

We see a landing page that is clearly too long, with a call-to-action that is buried under three layers of “corporate-speak,” and instead of fixing it, we “test” it against a version that is slightly less bad. We wait three weeks for the results. We pore over the heatmaps. We act like we are uncovering the secrets of the human psyche when we are really just confirming that people don’t like reading boring walls of text. It is a waste of intellectual capital.

“I recall a specific instance where I sat in a board meeting for a non-profit. They were debating whether to change the ‘Donate’ button from blue to orange. They had been running a test for three months. They had 120 visitors a week.”

– Internal Monologue of Sanity

The “data” suggested the orange button was winning by a margin of two donations. They spent forty-five minutes discussing the “psychology of orange” and how it evokes a sense of urgency. I sat there, “99 Red Balloons” still thumping in my ears, and realized we were all insane. We were spending thousands of dollars in billable hours to discuss the equivalent of a rounding error.

The “Small-n” Reality

The alternative is to embrace the “small-n” reality. If you have 900 visitors a month, your best tool is not a testing suite; it is a phone. Or a Zoom link. Or a recording of a single user struggling to find the “Contact” page.

One video of a person getting frustrated with your navigation is worth more than a year’s worth of insignificant p-values. It provides a liminal insight that numbers cannot touch-the “why” behind the “what.”

We have to stop pretending that every website is Amazon. Amazon can test the specific shade of blue on a link because they have millions of interactions an hour. They can see the signal because the noise is a tiny fraction of their volume. For the rest of us, the noise is the majority of the broadcast. When you have low traffic, your “experiments” are just anecdotes with a decimal point.

Admitting the Limits of Measurement

It takes courage to say, “We don’t have enough data to be sure, so we are going to make a judgment call based on our expertise.” It feels risky. It feels unscientific. But it is actually more honest than the alternative.

Admitting the limits of your measurement is the first step toward actual rigor. Otherwise, you are just an anchorite in a digital cell, worshipping a god that isn’t listening.

The next time you find yourself in a Wednesday stand-up, looking at a green bar that represents three more people than the red bar, take a breath. Ignore the dashboard for a second. Ask yourself: “If we didn’t have this software, what would we obviously do to make this page better?”

Don’t let the ceremony of the test become the work itself. Measurement is a tool, not a religion. When the volume is low, the best measurement is the one that happens between your ears, informed by experience and a genuine connection to the person on the other side of the screen.

Everything else is just expensive static.