Measuring a conversational programme without fooling yourself

Not engagement. Not reach. One event, already recorded elsewhere in your business, that you would be pleased to cause more of: an appointment attended, an invoice paid, a renewal completed, a ticket resolved without a second contact.

If you cannot name one, the programme does not have a purpose yet, and no measurement will supply one.

Randomly select a percentage of the eligible population and send them nothing. Measure the same outcome on both groups.

Everything else on this list is refinement. This is the measurement.

Three rules make it work:

• Randomise per person, not per segment. Segment-level splits import every difference between segments into your result.
• Keep it permanently. A holdout that runs for one quarter tells you about that quarter.
• Keep it small but not tiny. Large enough to detect the effect you care about; small enough that leadership does not object to the foregone revenue.

Sent, delivered, seen, replied, acted. Each stage as a rate on the previous stage, and the final outcome as a rate on sent rather than delivered.

Using delivered as the denominator is the most common way a report flatters itself, because it excludes exactly the failures you most need to see.

Outcome moves slowly. Three things move first:

• Unsubscribe rate per send. Rises before revenue falls.
• Reply sentiment. Cheap to sample by hand; a hundred replies read by a human beats a dashboard.
• Repeat contact rate. If people come back about the same thing, you contained rather than resolved.

Attribution windows are a choice, not a fact. Multi-touch models are assumptions wearing a chart. Channel comparisons across different audiences are not comparisons.

Reports that state their limits are trusted more, not less — and they survive the question "how do you know?" from someone senior, which is the actual test.

The most credible measurement report has a section listing what it does not establish. Attribution across channels, long-run brand effects, and anything happening to people who never opened the message all belong there.

Writing that section is not a weakness. It is the thing that makes the rest of the numbers believable to anyone who has seen a marketing deck before.

Well-run programmes routinely find that a meaningful share of messages have no measurable effect. That is not a failure of measurement; it is the finding. The right response is to send fewer, better-targeted messages — which lowers cost and raises the outcome rate at once.

The programmes that never find this are usually the ones without a holdout.

Delivery rate measures the network. Everything a business cares about happens after it, and each step below loses more than the one before.

Every number above is a level, and levels are not results. The programme did not cause 24 completions; it caused the difference between 24 and whatever would have happened anyway.

That difference is unknowable without a group who did not receive the message. There is no analytical substitute — not attribution windows, not a pre-period comparison, not a model. Those techniques estimate what a holdout measures.

The objections are always the same two, and both have answers.

"We are giving up revenue on the holdout." You are giving up a small, known amount in order to find out whether the rest is real. A programme that cannot survive a 5% holdout is a programme nobody has checked.

"Our segments are too different to randomise." Randomise within segment. The point of randomisation is that it balances the things you did not think to control for, which is most of them.

Revenue moves last. By the time it moves, the cause is a quarter old and the people who left are not coming back. Four indicators move first.

Build the holdout before you launch, keep it permanently, watch the four leading indicators weekly, and report the funnel rather than the delivery rate. Then write down what you still cannot prove, and resist the pressure to quietly drop that paragraph.