A/B testing Bitrix24 email campaigns: setting up accurate tests

A/B testing in Bitrix24: where statistics breaks

In Bitrix24, you can launch an A/B test of an email campaign in a couple of clicks — and that is exactly where most campaigns break.

Emails go to the entire base in two branches, and the difference between them is accumulated from different conditions: one gets active customers, the other gets those who have not opened emails for months.

Formally, the test has been run, but what is compared is not versions of the email, but two dissimilar audiences.

Next, segment overlap comes into play: the same address ends up in both branches because of overlapping filters and exclusion lists, and receives two emails in a row.

Alongside that, there is different delivery speed: the branches go out at different times, some addresses are still queued, while opens for the first one have already been counted. Any such detail shifts the result more than the email text itself.

Mistakes in test design are almost always the same: the hypothesis is not fixed, the subject line, send time, and call-to-action button are changed at once, there is no control group, and response is measured by opens, without separating real reads from automatic preview and image loading.

Together, this produces unreliable test conclusions — the winner is chosen based on random noise, not on the effect of the wording.

The outcome is predictable: email campaign response does not grow, but confidence that "A/B does not work for us" gets stronger.

Below, we break down how this is closed — from hypothesis formulation and clean audience splitting to calculating the required sample size and checking that the difference between branches is actually significant.

What a correct hypothesis testing methodology in Bitrix24 gives

Correct A/B testing starts with comparable groups. The audience is split not by order in the list, but by a stable attribute — contact ID or activity segment.

Then the branches match in share of active subscribers, date of last purchase, and acquisition channel, and the response shift relates to the hypothesis, not to the list composition.

The second pillar is database data cleanliness.

Undeliverable addresses, duplicates, unsubscribes, and corporate traps are cleaned before the run starts: otherwise, part of a branch will not receive the email, the denominator will shift, and the open rate will grow on paper with the same real response.

Controlled test branches mean one variable per run. If the subject line changes, the segment, send time, and template remain unchanged, so the cause of the shift is read unambiguously.

Result reproducibility is checked with a second run on another segment: if the win reproduces, it is a pattern, not a coincidence in a single sample.

To grow email campaign response, you need a pre-fixed win criterion: metric, group size, and significance threshold.

Then the conclusion is defended before management with numbers, not with the feeling "it seems to have gotten better." In Bitrix24, branches are tagged and compared with standard tools, and the send log serves as the evidence base when analyzing the result.

Test design options: continuous, segmented, paired

The launch scheme determines what you are actually comparing: the audience's reaction to the email text or the difference between two non-identical groups of recipients. If chosen incorrectly, it shows growth in the report that does not reproduce in the next email campaign.

Separately, we consider sample limitations. To draw conclusions on opens and clicks for each version, you need a reserve of active addresses — with several hundred recipients, a difference of one or two percent lies within random variation.

We exclude undelivered and unsubscribed from the denominator: otherwise, the versions have different bases, and result bias arises even before comparison.

In sequential waves, we additionally fix the day of week and hour of sending, so that the email effect does not mix with audience behavior over time.

Implementation stages: from database audit to regulations

Implementing A/B testing in Bitrix24 starts not with email setup, but with data inventory. If the database is not tagged and scenarios overlap, the test will show a difference between samples, not between hypotheses.

Therefore, the work proceeds sequentially — from database analysis to maintenance rules that remain with your team.

1. Audit of the database and scenarios. We check field completion, activity history, duplicates, and non-working addresses, and see how already running chains affect segments. The output is a data map suitable for building a sample.

2. Branch scheme. We fix one variable and one metric — email subject, offer block, or send time. Other parameters remain unchanged, otherwise the result cannot be attributed to a specific change.

3. Sample tagging. The database is divided into a control group and test groups, comparable by activity, source, and time in the database. This eliminates the skew that causes a "warmer" segment to "win."

4. Scenario setup. We build the chain in Bitrix24: entry conditions, pauses, exit points, substitutions from contact fields. Separately, we check that test branches do not overlap with transactional emails.

5. Control send. A run on a small test segment: rendering in email clients, link operation, substitution correctness, and recipients landing in the right branches. Only after that — launch across the entire sample.

6. Collecting results. We consolidate opens, clicks, and target actions, compare with the control group, and assess the significance of differences. We do not consider interim snapshots a basis for a decision.

7. Handover of regulations. We provide a document: which variables we test, how the sample is formed, minimum group size, observation period, and the procedure when results diverge.

This order gives reproducibility: the next test launches by the same rules without re-analyzing mistakes from scratch.

Segment schemes, email templates, and test stopping regulations

The segment, template, and stopping criterion described in correspondence are lost at the first change of the person responsible — and the next test launches anew, by intuition. To make the result reproducible, we fix the materials in the portal. Here is what remains in the system after testing is set up.

• Segment scheme. Contact, deal, or smart process fields from which the sample is assembled, inclusion logic, and behavior when status changes. The composition is fixed at the start — branches must not receive a different audience in the middle of the test.

• Email templates. Both versions are stored side by side and differ by one element: subject, first paragraph, or button. Layout and the rest of the text are shared — otherwise the growth cannot be explained by anything but a guess.

• Rotation rules. How the database is split: by a given share, by contact ID parity, or by source. Here also is the exclusion list — unsubscribed, repeat inquiries, open deals — so they do not enter the sample on the second round.

• Stopping regulations. Minimum sample size, observation period, difference threshold between branches, and the condition "no result detected." Separately — early stop on anomaly: a spike in unsubscribes, complaints, personalization failure.

• Test documentation. A log with the hypothesis, changed element, start date, owner, final metrics, and decision — scale or reject. It also becomes the basis for the next hypothesis.

• Roles and access. Who owns the segment, who approves the template, and who has the right to stop a branch early — so that the decision does not depend on one person.

The set remains in the portal, not in correspondence: a new responsible person opens the log, takes the ready segment scheme, and repeats the test on a comparable audience. Conditions are not reconstructed from memory — they lie next to the email campaign.

Case study: how stream separation eliminated a deliverability failure

The client is the marketing department of a service company with a base of about 40 thousand subscribers in Bitrix24. One general email campaign went to the entire base once a week, and the launch was done immediately after segment formation.

After two months, the share of complaints about the campaign grew, deliverability dropped: some emails stopped reaching inboxes, and unsubscribes came in a wave after each issue.

The analysis showed two causes. First — the stream was not separated: emails went at one pace and without considering subscriber activity, so the "sleeping" part of the base received the same issues as the active part.

Second — the schedule: sending fell into an overloaded window, emails went into the queue with a delay and lost relevance.

What was changed

We split the base into three streams by activity: those who opened emails in the last month, those who opened less often, and those who never showed activity.

For each stream, we set its own schedule and frequency — active ones kept the weekly issue, others were reduced to one every two weeks with a separate email subject.

Then we enabled A/B testing: within a stream, we split the sample into two groups and compared the email subject and send time, not the entire content at once.

For those who did not open, we set up one reminder scenario, after which the address went into the passive stream instead of repeated sends.

What happened

After six weeks, the share of complaints about the campaign fell by almost three times, unsubscribes — by about half, deliverability returned to a level at which the main part of the base receives emails in the inbox.

Conversion from open to click grew by a third for the active stream: the email subject was selected by a test, not by one person's hypothesis.

The passive stream stopped dragging overall metrics down, while base coverage fell only slightly — due to the fact that some "sleeping" subscribers returned after the frequency change.

Before and after metrics were taken from one dashboard to separate the effect of stream separation from seasonal fluctuations.

How does a paired test differ from sequential waves?

A paired test launches two versions of an email simultaneously, to two comparable groups of recipients, and they differ by exactly one element — subject, first paragraph, call-to-action button.

Both groups live in the same calendar window, so seasonal fluctuations — a sale, holidays, a spike in shipments — affect them equally and are subtracted in the difference.

The condition is one: groups are formed by random splitting and do not overlap, otherwise the same recipient will fall into both versions and group comparability is lost.

Sequential waves are structured differently: version A goes to the entire base in one period, version B — in the next. This is simpler when the base is small and there is nothing to split it by.

But seasonal fluctuations stop being a common background and become a source of error: the second wave goes under different conditions, and you are measuring not the email, but the difference between periods.

By default, we choose a paired test: it gives a conclusion that can be relied on for the next launch. We leave waves for cases when a change is evaluated as a whole — for example, a change in the tone of an entire email series.

In Bitrix24, both schemes are implementable; setup comes down to fixing the split and the rule by which the winner is determined before launch, not after looking at the numbers.

Engineer's doubts: sample size, overlaps, data cleanliness

Objections to A/B testing come down to four points: the sample size is small, segments overlap, automated chains are already running, and seasonal spikes and different send frequency interfere with comparison. Each is resolved by a rule before the start.

We count deliverable addresses after unsubscribes and bounces, check overlaps by contact key, and fix the variant label in the card. Comparison happens with the same send time and number of touches — the difference is made by the email text, not by the background.

Contacts fall into both segments — how to count?

One contact must not vote twice: we set rule priority, the variant label is stored in the card and transferred to deals, and repeated segments are cut off by a filter. If the overlap cannot be excluded, we compare groups within it separately.

Automated chains, seasonality, and send frequency?

We test the step after which the chain does not diverge: if previous emails differed by group, the final is explained by them. We compare seasonal spikes week to week, and split variants randomly within one wave.

The frequency of touches is the same for both groups — otherwise we measure the number of sends, not the text.

Let's discuss your task: what to prepare for the conversation

A substantive conversation about A/B testing email campaigns in Bitrix24 takes half an hour instead of three meetings if the inputs are collected in advance. Here is what is worth having at hand.

• Portal access. An account with rights to the email campaigns section and email builder plus a test environment or a copy of the working portal — we run variants there without touching live databases.

• List of scenarios. Which emails and chains are already running, which we compare with each other and at which step — welcome, abandoned cart, repeat sale.

• Result metric. What we use to determine the winner: opens, clicks, replies, submitted requests. One test — one metric, otherwise the conclusion will not add up.

• Email campaign history. Statistics for several periods, segment sizes, and rules for assembling them — from this we calculate the required sample size.

• Constraints. Send frequency, subscriber consents, design requirements — everything that must not be violated in any version.

• Expectations for the result. What success means to you: growth in response at a specific step, fewer unsubscribes, a fuller base. This goes into the testing plan.

With this set, we will come back with an estimate and work plan: how many variants make sense to test, which segments to take, and when the result will be visible.