Email Measurement
Part of Subject line and email content experiments
Recording campaign learnings beyond the winning variant
Capture the question, variants, audience, raw outcomes and limits of an email test so the next campaign can use the learning.
Keep the test question, exact variants, audience, counting rules and observed results together. A note reading only “subject B won” cannot tell a future editor whether the finding applies to another campaign or whether a tracking problem shaped it.
Preserve what was sent
Save the final subject and preview text, content versions, destinations and send window. Record the eligibility rule, exclusions and numbers assigned to each version if available. If the campaign changed after launch, record what changed and when. A screenshot or exported report can preserve the platform view, but add a short explanation of the test.
The record should answer five questions:
- What choice was the team trying to make?
- What differed between the versions, and what stayed steady?
- Which outcome and observation window were chosen in advance?
- What did each version produce, including raw counts?
- What decision followed, and how far can it be applied?
Keep the evidence behind the label
For each version, record the number assigned, the number successfully delivered and the number of people completing the primary action when those figures are available. Name the denominator before adding a rate.
Keep bounces, unsubscribes and measurement issues near the result. If opens were examined, note possible privacy-related distortion. If clicks were central, record the available bot-filtering setting and any known tracking limits.
Do not assume a campaign summary export contains every detail needed to reproduce a calculation. Keep definitions and any separately authorised analysis with the record. Aggregate results usually serve an editorial learning note without copying personal subscriber data into it.
Write a bounded learning
An illustrative note might say: “For this workshop invitation, subject B led on recorded clicks to the booking page during the agreed window. The difference was small, so both subjects remain candidates.” It describes no real test result. It preserves context and explains why no permanent rule followed.
Separate observation from interpretation. Observation reports counts. Interpretation may suggest that a clearer promise helped, but that remains a hypothesis unless the test isolated it. Record surprises such as uneven delivery, a broken destination or a lead appearing only on a secondary measure.
State the next use
End with an action: reuse a version for a closely similar message, repeat the comparison, revise both versions or change the measurement plan. Add an owner and a condition for revisiting the decision. Attach later, conflicting results to the earlier record so the learning history remains traceable.
