Email Measurement
Part of Subject line and email content experiments
Choosing a useful success metric for an email test
Select a primary email-test outcome, define its numerator and denominator, and keep tracking limitations close to the decision.
Choose one primary metric that answers the test question. Before sending, define the action, who is counted and the observation window. A convenient dashboard number may not measure the decision the campaign is meant to support.
Work backwards from the email’s job
If the email offers a guide, a tracked click on its link may show progress towards that guide. If it invites an appointment, a completed booking is closer to the goal, provided the booking can be attributed reliably to the assigned variant. Opens cannot show whether changed body copy helped because the reader opens the email before seeing that copy.
| Email’s job | Candidate primary metric | Check before using it |
|---|---|---|
| Direct readers to a guide | Unique people clicking the guide link | Is the link tracked, and could automated clicks affect the figure? |
| Generate enquiries | Completed enquiry forms | Can each completion be attributed to an assigned variant? |
| Prompt a booking | Completed bookings | Are duplicates and cancellations handled consistently? |
A click does not prove that a page loaded or a booking happened. When the final action cannot be measured credibly, use an earlier step and name it accurately.
Match the email’s job to a candidate primary metric
- Direct readers to a guidePrimary metric: Unique people clicking the guide link. Check: Is the link tracked, and could automated clicks affect the figure?
- Generate enquiriesPrimary metric: Completed enquiry forms. Check: Can each completion be attributed to an assigned variant?
- Prompt a bookingPrimary metric: Completed bookings. Check: Are duplicates and cancellations handled consistently?
Define the counting rule
State whether the numerator is people who acted or total actions. Repeated clicks by one person should not quietly become several successful readers. State the denominator too. Mailchimp reports include click rates; confirm how your platform defines the denominator.
For a randomised test, keep the original number assigned to each version and check delivery differences. Excluding more people from one version after assignment can complicate the comparison.
Choose the observation window in advance and apply the same rule to both versions. A later booking can remain commercially relevant, but report it separately from the planned test window.
Define the counting rule before sending
- State whether the numerator is people who acted or total actions
- State the denominator and confirm how the platform defines it
- Keep the original number assigned to each version for a randomised test
- Check delivery differences between versions
- Choose the observation window in advance and apply it to both versions
- Report later bookings separately from the planned test window
Use other measures as checks
Keep bounces, unsubscribes and complaints visible for possible adverse effects. Opens can provide context, but privacy-related pixel loading and blocked images limit their interpretation.
Automated security systems can affect clicks too. Check the platform’s available filtering without assuming it removes every automated action.
A purchase or revenue measure needs a clear attribution rule connecting recipient, variant and order. If that connection cannot be explained, it should not become the primary test outcome.
Use other measures as checks
- Keep bounces visible for possible adverse effects
- Keep unsubscribes visible for possible adverse effects
- Keep complaints visible for possible adverse effects
- Use opens only as context; privacy-related pixel loading and blocked images limit interpretation
- Check the platform’s available filtering for automated clicks without assuming it removes every automated action
- Use a purchase or revenue measure only with a clear attribution rule connecting recipient, variant and order
Set the decision rule
Before sending, complete this sentence: “We will prefer the variant with more [defined actions] per [defined group] by [time], provided [guardrail] remains acceptable.” Save it with the variant definitions.
Report secondary measures as context rather than replacing the planned outcome after the results arrive.
Set the decision rule
- Before sending, complete the decision sentenceWe will prefer the variant with more [defined actions] per [defined group] by [time], provided [guardrail] remains acceptable.
- Save it with the variant definitions
- Report secondary measures as context
- Do not replace the planned outcome after the results arrive
