---
title: "Measuring a conversational programme without fooling yourself"
summary: "A method: choose one outcome, build a permanent holdout, instrument the funnel, and be honest about what the numbers cannot settle."
canonical: https://cpaas.co/en/learn/measuring-a-conversational-programme
kind: whitepaper
depth: applied
origin: original
published: 2026-08-22T16:26:18.899Z
updated: 2026-10-14T16:26:19.485Z
topics: ["Analytics", "Conversational Marketing", "Customer Experience"]
language: en
---
# Measuring a conversational programme without fooling yourself

Most messaging reports measure the network and call it a result. Delivery rates in the high nineties, open rates that look healthy, a revenue figure attributed to "customers who received a message" — and no way to say how many of those customers would have done the same thing anyway. When someone senior asks "how do you know it worked?", the honest answer is often "we don't". This piece is a five-step method that produces a better answer, and a worked example of the arithmetic.

## Step one: name the outcome, singular

Not engagement. Not reach. One event, already recorded elsewhere in your business, that you would be pleased to cause more of: an appointment attended, an invoice paid, a renewal completed, a ticket resolved without a second contact.

If you cannot name one, the programme does not have a purpose yet, and no measurement will supply one.

## Step two: build the holdout before you launch

Randomly select a share of the eligible population and send them nothing. Measure the same outcome in both groups. Everything else in this piece is refinement; this is the measurement.

- **Randomise per person, not per segment.** Segment-level splits import every difference between segments into the result. If segments differ a lot, randomise within each one.
- **Keep it permanently.** A holdout that runs for one quarter tells you about that quarter.
- **Size it to the effect you care about.** Large enough to detect a difference worth acting on; small enough that the foregone revenue is accepted. Five to ten per cent is a common starting point for large programmes; small programmes need a larger share or a longer run.

The usual objections have answers. *"We are giving up revenue."* A small, known amount, to find out whether the rest is real. *"Our segments are too different."* Randomise within segment; randomisation balances the things you did not think to control for, which is most of them.

## Step three: instrument the funnel

> **Interactive figure — delivery-funnel.** Set the stages to your own numbers. The last bar is the programme; the first three explain it.
> Available at https://cpaas.co/en/learn/measuring-a-conversational-programme

Sent, delivered, seen, replied, acted. Show each stage as a rate on the *previous* stage, and the final outcome as a rate on **sent**, not delivered. Using delivered as the denominator is the most common way a report flatters itself, because it removes exactly the failures you most need to see.

```diagram:bars
# An illustrative outbound funnel, per thousand sent
Delivered | 970 | The network did its job. This is where most reporting stops.
Opened or read | 620 | Where a read receipt exists. On SMS you are inferring this, not measuring it.
Clicked or replied | 71 | The first step that requires a decision by a person.
Completed the action | 24 | Bought, booked, renewed, confirmed. The only column finance recognises.
Still there next quarter | 19 | Net of the people this send cost you.
> The shape is the point, not the numbers. The steepest drop is never between sent and delivered — which is the only gap most dashboards show.
```

Note that "seen" means different things per channel: WhatsApp and RCS return read receipts (which users can switch off); SMS returns none. Never average across channels as if it were one measurement.

## Step four: watch the leading indicators

Revenue moves last. By the time it moves, the cause is months old. These move first:

```diagram:flow
# The order damage shows up in
Unsubscribe rate rises | Per send, not per month. The earliest honest signal, and the cheapest to watch.
Reply sentiment sours | Sample a hundred replies by hand. A human reading them beats any dashboard at this size.
Repeat contact climbs | People coming back about the same thing means you contained rather than resolved.
Engagement decays | Opens and clicks fall on the same audience for the same content.
Revenue falls | Last, and by now the cause is months behind you.
> Anything you can only detect at the last step is something you found out too late to act on.
```

## Step five: say what you cannot know

Attribution windows are a choice, not a fact. Multi-touch models are assumptions wearing a chart. Channel comparisons across different audiences are not comparisons. A credible report has a short section listing what it does not establish — cross-channel attribution, long-run brand effects, effects on people who never opened the message. That section is what makes the rest believable.

```diagram:checklist
# The section that makes a report believable
[ ] Cross-channel attribution | Not established by this design
[ ] Long-run brand effects | Not established
[ ] Effects on people who never opened the message | Not established
> Listing what the report does not settle is what lets the rest be believed.
```

## A worked example

A utility company sends a WhatsApp or SMS payment reminder three days before a bill is due.

- **Eligible:** 100,000 customers. **Holdout:** 5,000, chosen at random, receive nothing. **Messaged:** 95,000.
- **Outcome:** bill paid on time.
- **Result:** 88.0% of the messaged group paid on time; 85.5% of the holdout did.

- **Lift:** 2.5 percentage points, or about 2,375 extra on-time payments (95,000 × 0.025).
- **Cost:** 95,000 messages. If each costs a few cents, the programme's real price is the total divided by 2,375 — not by 95,000, and certainly not by the 83,600 who paid.

```diagram:stats
# Paid on time: messaged against holdout
88.0% | Messaged group | 95,000 customers
85.5% | Holdout | 5,000 customers, chosen at random, sent nothing
> Two nearly equal towers. The reminder's effect is the gap between them — 2.5 points, about 2,375 payments — not the height of either.
```

The finding that matters most is the uncomfortable one: most customers who paid would have paid anyway. That is not a failure of measurement; it is the measurement. It usually leads to sending fewer, better-targeted reminders — for instance only to customers who paid late before — which lowers cost and raises the effect per message.

## What to ask a provider

- Can I export every message event (sent, delivered, read, replied, failed) with my own customer ID and timestamps?
- What exactly does "delivered" mean on each channel, and when is a read receipt unavailable?
- Can your platform hold out a random share of an audience, and is the selection logged per person?
- Will you report outcomes against sends, or only delivery and click rates?

## Where to go next

The funnel argument starts in [delivery rate is not a result](/en/learn/delivery-rate-is-not-a-result). For how to read other people's numbers with the same scepticism, read [what analysts disagree about](/en/learn/what-analysts-disagree-about); to build measurement rights into an agreement, see [evaluating a CPaaS contract](/en/learn/evaluating-a-cpaas-contract).

---

Published by the company behind five products in this category. Those products appear in this site's directory alongside competitors under the same published criteria and are labelled as its own. Editorial content does not recommend them.