Measuring a conversational programme without fooling yourself

Most messaging reports measure the network and call it a result. Delivery rates in the high nineties, open rates that look healthy, a revenue figure attributed to "customers who received a message" — and no way to say how many of those customers would have done the same thing anyway. When someone senior asks "how do you know it worked?", the honest answer is often "we don't". This piece is a five-step method that produces a better answer, and a worked example of the arithmetic.

Not engagement. Not reach. One event, already recorded elsewhere in your business, that you would be pleased to cause more of: an appointment attended, an invoice paid, a renewal completed, a ticket resolved without a second contact.

If you cannot name one, the programme does not have a purpose yet, and no measurement will supply one.

Randomly select a share of the eligible population and send them nothing. Measure the same outcome in both groups. Everything else in this piece is refinement; this is the measurement.

• Randomise per person, not per segment. Segment-level splits import every difference between segments into the result. If segments differ a lot, randomise within each one.
• Keep it permanently. A holdout that runs for one quarter tells you about that quarter.
• Size it to the effect you care about. Large enough to detect a difference worth acting on; small enough that the foregone revenue is accepted. Five to ten per cent is a common starting point for large programmes; small programmes need a larger share or a longer run.

The usual objections have answers. "We are giving up revenue." A small, known amount, to find out whether the rest is real. "Our segments are too different." Randomise within segment; randomisation balances the things you did not think to control for, which is most of them.

Sent, delivered, seen, replied, acted. Show each stage as a rate on the previous stage, and the final outcome as a rate on sent, not delivered. Using delivered as the denominator is the most common way a report flatters itself, because it removes exactly the failures you most need to see.

Note that "seen" means different things per channel: WhatsApp and RCS return read receipts (which users can switch off); SMS returns none. Never average across channels as if it were one measurement.

Revenue moves last. By the time it moves, the cause is months old. These move first:

Attribution windows are a choice, not a fact. Multi-touch models are assumptions wearing a chart. Channel comparisons across different audiences are not comparisons. A credible report has a short section listing what it does not establish — cross-channel attribution, long-run brand effects, effects on people who never opened the message. That section is what makes the rest believable.

A utility company sends a WhatsApp or SMS payment reminder three days before a bill is due.

• Eligible: 100,000 customers. Holdout: 5,000, chosen at random, receive nothing. Messaged: 95,000.
• Outcome: bill paid on time.
• Result: 88.0% of the messaged group paid on time; 85.5% of the holdout did.

• Lift: 2.5 percentage points, or about 2,375 extra on-time payments (95,000 × 0.025).
• Cost: 95,000 messages. If each costs a few cents, the programme's real price is the total divided by 2,375 — not by 95,000, and certainly not by the 83,600 who paid.

The finding that matters most is the uncomfortable one: most customers who paid would have paid anyway. That is not a failure of measurement; it is the measurement. It usually leads to sending fewer, better-targeted reminders — for instance only to customers who paid late before — which lowers cost and raises the effect per message.

• Can I export every message event (sent, delivered, read, replied, failed) with my own customer ID and timestamps?
• What exactly does "delivered" mean on each channel, and when is a read receipt unavailable?
• Can your platform hold out a random share of an audience, and is the selection logged per person?
• Will you report outcomes against sends, or only delivery and click rates?

The funnel argument starts in delivery rate is not a result. For how to read other people's numbers with the same scepticism, read what analysts disagree about; to build measurement rights into an agreement, see evaluating a CPaaS contract.