A/B Testing Your Email Campaigns: What Works in Braze

Braze email A/B testing compares versions of one email on a random slice of your audience, then sends the best performer to the remaining users. The result is only as sound as the setup behind it: the metric you pick, the audience you filter and what you decide before sending. This guide covers the mechanics, the limits and the decisions to make first.

‍

Key Takeaways

• Braze email tests support up to eight variants, and tests with more variants need a larger test group.
• Braze says you likely need around 15,000 users per variant, including the control, to reach 95% confidence.
• Changing the subject line or HTML body after launch compromises the experiment, and Braze disables it.
• Judge a test with a control group on a conversion event, because Braze doesn't recommend deciding by opens or clicks.
• The Winner label can appear without statistical significance if you chose to send the best performer anyway.
• Write five decisions down before the first send, including what happens if no variant reaches 95% confidence.

‍

How does Braze email A/B testing work?

Braze describes an A/B test as one that examines the effect of changing one variable, and a multivariate test as one that examines two or more. You can create up to eight variants of your message, and you can test any campaign that targets a single channel.

In a single-send test, Braze sends an initial portion of the audience to each variant. It then sends the best-performing variant to the remaining users. A user who receives the campaign again gets the same variant.

For email, Braze lists the things to vary: subject line, display name, salutation, body copy, image and emoji usage, and how numbers and times are presented. We recommend changing one of them per test, unless your audience is large enough for a multivariate design.

‍

How do you set up an email A/B test in Braze?

Braze lets you reserve a percentage of your target audience for a randomised control group. You can also set a conversion event, to see how many recipients performed a particular action. We recommend working through the setup in this order, because the last step is the one teams break.

1. Create a campaign and choose the email channel.
2. Compose your variants, changing only the element you want to test.
3. Schedule the campaign the way you'd schedule any other.
4. Choose a segment, then split it across the variants and an optional control group.
5. Designate a conversion event if the test will be judged on conversion.
6. Review and launch, then leave the message alone.

‍

After a test begins, Braze considers the experiment compromised and disables it if you change parameters such as the subject line or HTML body. Braze also recommends avoiding message edits within an hour of the experiment campaign launch.

Here's a worked example of the sequence. Say you want to compare two subject lines on a recurring newsletter. Change only the subject line, keep the display name and body fixed, filter the segment to subscribed users, and name a conversion event before launch. Then write down which result would make you change the subject line, so the decision exists before the data does.

‍

How large does the test audience need to be?

Braze gives a guide rather than a threshold. It says you likely need around 15,000 users per variant, including the control, to achieve 95% confidence, and it references a sample size calculator.

Braze also notes that multivariate and A/B tests with more variants require a larger test group to reach statistically significant results. We read the cap of eight variants as a ceiling for very large audiences, not a target.

Braze adds that the success of a message says something about both the message and its target segment. We wouldn't carry a win from one segment straight into another without a fresh test. A subject line that works for loyal customers may say little about lapsed ones, so treat each segment as its own question.

‍

Which metric should decide the winner?

Braze lists opens and conversion rate as the results to look for in an email test. Under Optimize with BrazeAI, the default goal for email is unique clicks.

Control groups change the answer. Braze states that using a control group when determining a winner by opens or clicks isn't recommended. Control users don't receive the test, so they have no message to open or click.

If you hold out a control, we recommend judging the test on a conversion event and treating opens as a diagnostic. If your conversion event is a custom event, agree its name before you start, so that reports stay readable.

‍

What happens when no variant wins?

Braze tests all the variants against each other with Pearson's chi-squared tests, and it calls a result significant at the 95% level. That's a separate test from the confidence score, which only describes a variant's performance against the control on a scale of 0 to 100%.

The detail to watch is in the analytics documentation. If no variant beats all the others with 95% confidence and you chose to send the best-performing variant anyway, that variant is still sent. It also carries the Winner label.

So the label can appear without statistical significance. We recommend deciding in advance whether a non-significant result should trigger the second send. If the answer is no, don't choose the send-anyway option.

‍

Should you let Optimize with BrazeAI pick the winner?

Optimize with BrazeAI is the automatic route, and your campaign must include at least two message variants. It behaves differently depending on whether the campaign sends once or repeatedly.

‍

For the experiment duration, you can select 4 hours, 24 hours, 72 hours or a custom duration. The default is 4 hours, or 24 hours if you optimise for a primary conversion event.

Recurring, action-based and API-triggered campaigns that send multiple times count as multi-send. On a multi-send campaign, you can add or remove a control group under advanced controls. Braze describes it as a baseline for measuring campaign performance.

The email default of unique clicks is Braze's starting point. We recommend choosing the goal that matches the outcome your programme actually cares about.

‍

Which mistakes spoil a Braze email test?

The first is an uneven audience. Braze's own example is push, where the initial audience must filter for having a push token. Braze names opted in, has a push token and subscribed as the kinds of eligibility to filter on. For email, that means filtering for subscribed users before the split, so every test and control user could have received the message.

The second is expecting a control user to join the audience later. Braze says users who previously received messages can't enter the control group on a later send, and no user in the control group can ever receive a message. We recommend settling the control group size before the first send and keeping it fixed for a recurring test.

The third is rate limiting, which Braze describes in two ways. The create page says the rate limit isn't applied to the control group in the same way as the test group, which is a potential source of time bias. The FAQ says Braze currently doesn't support rate limiting with an A/B test that has a control group. We suggest planning on the stricter reading, and checking the campaign builder before you rely on either.

Finally, give an email test time. Braze notes that pushes show results faster because users see them immediately, but it may be days before they see or open an email.

‍

What changed recently?

Braze's 25 Jun 2026 release notes list Optimize with BrazeAI as early access, and they note that it turns on automatically when you add multiple push variants. Availability can differ by account, so we recommend confirming it with your Braze contact before you plan around it.

On the same date, Quick Push A/B Testing reached general availability, with support for multi-platform push campaigns and Canvas steps through variant groups. It's a push feature, so it doesn't change the email steps above.

‍

What should you decide before the first send?

As a Braze implementation partner, CustomerIK recommends writing five decisions down before an email test goes out.

1. The one thing you're changing, and the goal or conversion event that will judge it.
2. The audience filter that makes every test and control user eligible.
3. The audience size, checked against the guide of 15,000 users per variant.
4. Whether you'll run a single send or optimise a recurring campaign.
5. What happens if no variant reaches 95% confidence.

‍

Agreed in advance, those five lines make the result something you can act on instead of argue about. One failure mode is a test judged on opens with a control group attached, because the control can never open anything. Another is a send-anyway setting nobody remembers choosing, which puts a Winner label on a result that isn't significant.

‍

How does this connect to the rest of the programme?

Read test results alongside email campaign reporting, and agree conversion event names using the approach in user event tracking. If your tests depend on inbox placement, email deliverability setup comes first.

Once you have a winner, carry it into high-converting email campaigns. It also gives you a baseline for win-back campaigns.

‍

Frequently Asked Questions
‍

1. How many variants can a Braze email test have?

Up to eight variants of your message. Braze says tests with more variants require a larger test group to reach statistically significant results.

2. Can I edit an email after the test starts?

Braze considers the experiment compromised and disables it if you change parameters such as the subject line or HTML body. Braze recommends avoiding message edits within an hour of launch.

3. How does Braze pick the winner?

Braze tests all the variants against each other with Pearson's chi-squared tests and looks for one that outperforms the rest at 95% significance. If none does and you chose to send the best performer anyway, the Winner label can still appear.

4. Should I use a control group when judging by opens?

Braze says it isn't recommended, because control users don't receive the test. Judge the test on a conversion event instead.

5. What does Optimize with BrazeAI need on a recurring campaign?

At least two variants, at least one conversion event, and a re-eligibility window of 24 hours or longer.

‍

Sources

• Braze, Create multivariate and A/B tests. Retrieved 06 Oct 2026.
• Braze, Optimize A/B tests with BrazeAI. Retrieved 06 Oct 2026.
• Braze, Multivariate and A/B test analytics. Retrieved 06 Oct 2026.
• Braze, Multivariate and A/B test FAQ. Retrieved 06 Oct 2026.
• Braze, Release notes for 25 Jun 2026. Retrieved 06 Oct 2026.
• Written by CustomerIK, a Braze implementation partner.

‍

CustomerIK is a Braze implementation partner working across onboarding, technical integration, marketing operations and customer data management.

If your email tests end in arguments about whether the winner was real, let's talk.

‍