ℹ️ This article is specific to the New Agillic Content Designer. Information provided may not apply to previous versions of the Agillic Content Designer. If your solution hasn't been migrated to Agillic New Content Designer, please refer to this other article: Understanding a Subject Line Test.
A Subject Line Test allows you to evaluate two or more subject line variations.
A Subject Line Test enables you to test which of two or more subject lines perform the best for a percentage of your recipients. Then, Agillic will automatically send the best-performing subject line to the remaining recipients in a Target Group after a set amount of time has passed.
To set up a functional Subject Line Test, you will need to set up some logic in your email. In the Email, you can set up two or more subject line variations. When creating more than two subject line variations the subject line test settings will be visible in the right-hand side block panel. Here you will be able to set up the percentage of recipients participating in the test period, how long the test should run, and how to find the winning subject line.
For your Subject Line test to work, you will need to have your Email Step as the first step of your Flow.
Warning: Don't fill in 100 % for the 'Ratio of users in Test'. If you do, the Subject Line Test will test for 100% of your recipients. For example, in a Target Group of 100, it would mean sending subject line 1 to 50 recipients and subject line 2 to 50 recipients. Therefore, there would be no additional recipients to send the best-performing subject line to.
Apple’s Privacy Update Impact on Subject Line Test
In autumn 2021 Apple will introduce changes to the Mail application to increase users’ control over their privacy. You can read more about what is being updated and what it means here.
The update will affect split subject line testing in Agillic. By default, subject line testing is calculated on open rates, but can also be tested on click or open/click rates.
In theory, a valid AB test can be completed with unreliable open rate data. In a large test pool, the spread of Mail app users should be about 50/50, so unreliable open rate data will be distributed equally. However, there is no way to ensure an equal spread of Mail app users between the two groups, so the data can not be reliably compared. The larger the test pool, the more reliable the data will be.