---
title: "Measuring a conversational programme without fooling yourself"
summary: "A method: choose one outcome, build a permanent holdout, instrument the funnel, and be honest about what the numbers cannot settle."
canonical: https://cpaas.co/en/insights/measuring-a-conversational-programme
kind: whitepaper
depth: applied
origin: original
published: 2026-07-17T11:34:07.986Z
updated: 2026-09-08T11:34:08.134Z
topics: ["Analytics", "Conversational Marketing", "Customer Experience"]
language: en
---
# Measuring a conversational programme without fooling yourself

## Step one: name the outcome, singular

Not engagement. Not reach. One event, already recorded elsewhere in your business, that you would be pleased to cause more of: an appointment attended, an invoice paid, a renewal completed, a ticket resolved without a second contact.

If you cannot name one, the programme does not have a purpose yet, and no measurement will supply one.

## Step two: build the holdout before you launch

Randomly select a percentage of the eligible population and send them nothing. Measure the same outcome on both groups.

Everything else on this list is refinement. This is the measurement.

Three rules make it work:
- **Randomise per person, not per segment.** Segment-level splits import every difference between segments into your result.
- **Keep it permanently.** A holdout that runs for one quarter tells you about that quarter.
- **Keep it small but not tiny.** Large enough to detect the effect you care about; small enough that leadership does not object to the foregone revenue.

## Step three: instrument the funnel

> **Interactive figure — delivery-funnel.** Set the stages to your own numbers. The last bar is the programme; the first three explain it.
> Available at https://cpaas.co/en/insights/measuring-a-conversational-programme

Sent, delivered, seen, replied, acted. Each stage as a rate on the *previous* stage, and the final outcome as a rate on **sent** rather than delivered.

Using delivered as the denominator is the most common way a report flatters itself, because it excludes exactly the failures you most need to see.

## Step four: watch the leading indicators

Outcome moves slowly. Three things move first:

- **Unsubscribe rate per send.** Rises before revenue falls.
- **Reply sentiment.** Cheap to sample by hand; a hundred replies read by a human beats a dashboard.
- **Repeat contact rate.** If people come back about the same thing, you contained rather than resolved.

## Step five: say what you cannot know

Attribution windows are a choice, not a fact. Multi-touch models are assumptions wearing a chart. Channel comparisons across different audiences are not comparisons.

Reports that state their limits are trusted more, not less — and they survive the question "how do you know?" from someone senior, which is the actual test.

The most credible measurement report has a section listing what it does not establish. Attribution across channels, long-run brand effects, and anything happening to people who never opened the message all belong there.

Writing that section is not a weakness. It is the thing that makes the rest of the numbers believable to anyone who has seen a marketing deck before.

## The uncomfortable finding to expect

Well-run programmes routinely find that a meaningful share of messages have no measurable effect. That is not a failure of measurement; it is the finding. The right response is to send fewer, better-targeted messages — which lowers cost and raises the outcome rate at once.

The programmes that never find this are usually the ones without a holdout.

## The funnel, and where it actually leaks

Delivery rate measures the network. Everything a business cares about happens after it, and each step below loses more than the one before.

```diagram:bars
# A representative outbound funnel, per thousand sent
Delivered | 970 | The network did its job. This is where most reporting stops.
Opened or read | 620 | Where a read receipt exists. On SMS you are inferring this, not measuring it.
Clicked or replied | 71 | The first step that requires a decision by a person.
Completed the action | 24 | Bought, booked, renewed, confirmed. The only column finance recognises.
Still there next quarter | 19 | Net of the people this send cost you.
> The shape is the point, not the numbers. The steepest drop is never between sent and delivered — which is the only gap most dashboards show.
```

## Why the holdout is the whole method

Every number above is a level, and levels are not results. The programme did not cause 24 completions; it caused the difference between 24 and whatever would have happened anyway.

That difference is unknowable without a group who did not receive the message. There is no analytical substitute — not attribution windows, not a pre-period comparison, not a model. Those techniques estimate what a holdout measures.

The objections are always the same two, and both have answers.

*"We are giving up revenue on the holdout."* You are giving up a small, known amount in order to find out whether the rest is real. A programme that cannot survive a 5% holdout is a programme nobody has checked.

*"Our segments are too different to randomise."* Randomise within segment. The point of randomisation is that it balances the things you did not think to control for, which is most of them.

## Leading indicators, and the lag they buy you

Revenue moves last. By the time it moves, the cause is a quarter old and the people who left are not coming back. Four indicators move first.

```diagram:flow
# The order damage shows up in
Unsubscribe rate rises | Per send, not per month. The earliest honest signal, and the cheapest to watch.
Reply sentiment sours | Sample a hundred replies by hand. A human reading them beats any dashboard at this size.
Repeat contact climbs | People coming back about the same thing means you contained rather than resolved.
Engagement decays | Opens and clicks fall on the same audience for the same content.
Revenue falls | Last, and by now the cause is three months behind you.
> Anything you can only detect at step five is something you found out too late to act on.
```

## What to take away

Build the holdout before you launch, keep it permanently, watch the four leading indicators weekly, and report the funnel rather than the delivery rate. Then write down what you still cannot prove, and resist the pressure to quietly drop that paragraph.

---

Published by the company behind five products in this category. Those products appear in this site's directory alongside competitors under the same published criteria and are labelled as its own. Editorial content does not recommend them.