Email Copy
Part of Subject line and email content experiments
Testing a subject line without changing the audience
Hold eligibility, timing and email content steady when comparing subject lines, then report the result with its limits.
Isolate a subject-line change: define one eligible audience, randomly assign its members to two versions and send both in the same window. Keep the message, sender, preview text and destination steady. A different audience or message would make the result harder to attribute to the subject.
Lock the eligible audience first
Write the inclusion rule before building variants. For example: “Subscribers eligible for this month’s workshop invitation.” Apply the usual permission and suppression rules. Putting recent customers in one version and prospective customers in the other would compare audiences as well as subjects.
Randomly assign people within the eligible group. Check the numbers assigned to each version and that nobody can receive both. If an important audience characteristic is uneven by chance, record it. A later, larger test could balance that characteristic while randomising within each group.
Test a meaningful difference
For a hypothetical spring workshop invitation, the subjects might be “Spring workshop dates are open” and “Choose your spring workshop date”. Both accurately describe the same offer. One emphasises availability; the other suggests a next step.
Save the exact wording, including punctuation and personalisation. Keep the preview text, From name, body, links, offer and send window the same. If one subject needs a different body to make its promise true, label the test as a broader message comparison.
Decide what better means
An open-rate lead can be misleading. Apple Mail Privacy Protection can preload tracking images, and image blocking can leave real reading unrecorded. A subject might also attract curiosity without helping people take the intended action.
Choose a primary outcome before sending, such as unique people clicking the workshop link. Completed bookings are closer to the goal if they can be attributed reliably to the assigned version.
Set the observation window and counting rule in advance. Keep assigned and delivered counts, bounces and unsubscribes alongside the result. Report the planned metric first; treat other figures as diagnostics.
Record a bounded conclusion
A useful conclusion is specific: “For this invitation and eligible audience, subject B produced more recorded clicks on the workshop link during the agreed window.” Add counts, denominators and measurement limits. Clicks do not prove page visits or bookings, and one campaign does not establish a permanent preference for a wording style.
If the difference is small or tracking is doubtful, record the result as inconclusive and repeat the question when another suitable send arises.
