How simMachines' Beyond A/B Leverages One-To-One A/B Measurement to Drive Incremental Uplift
Marketing campaigns rely on response modeling, conversion modeling and uplift modeling to try and get at the most efficient way to optimize return on marketing investment (ROMI). Uplift modeling specifically, is an approach that identifies a target audience that will purchase only if they receive an ad treatment and uses data from A/B tests to evaluate treatment response vs. control response to isolate this audience. This is typically done at a stratified sample or group level.
simMachines’ one-to-one A/B measurement application, Beyond A/B, goes beyond traditional uplift modeling at a group level, to uncover subsets of individuals where the test treatment had an incremental effect, along with the key predictive factors in each of those subsets. This lift can be measured as binary or based on incremental ad effect, and it is achieved by analyzing the behavior of “twin pairs” between test and control groups, at a one-to-one level using similarity-based machine learning technology. The precision of the targeting models based on this data, and the insights that result from it in the form of sub-segments of positive incremental lift, is a new level of audience behavior measurement that is highly granular and powerful in terms of achieving maximum results.
THE CUMULATIVE EFFECT OF BEYOND A/B ON ROMI
First, it’s important to consider the ultimate objective of marketing campaigns, which is to optimize yield per marketing dollar spent by isolating the target audience likely to be affected by a specific promotional treatment, and maximizing sales per campaign by reaching the audience in the most efficient way possible. Beyond A/B has multiple layers of impact outlined as follows:
IMPACT #1: ONE-TO-ONE PAIRING VS. GROUP LEVEL ANALYSIS
Beyond A/B’s first area of impact is in revealing the difference between test vs. control individuals at a one-to-one level who were part of a completed A/B test, and analyzing the difference in behavior between the two, then segmenting those individuals into groups from which lookalike audiences can be generated for higher yield marketing campaigns.
From a performance perspective, this one-to-one twin pairing creates a level of precision that goes beyond traditional group-level analysis. Stratified samples can only get you so far as they analyze the overall behavior of treatment vs. control at a group level. Often this yields a comparative analysis that shows the performance of test vs. control groups is similar. But what about positive ad responders where the treatment had a big effect that are within the broader test group? These individuals are hidden from view unless you can isolate their individual behavior.
One-to-One
Paring Impact:
~ 5-20% + Lift in
Targeting Efficiency
Individual level response behavior yields a powerful training data set for a machine learning model looking to learn who is likely to be positively affected by a treatment. The bottom line is that the one-to-one level precision of Beyond A/B provides the foundation for a performance lift of 5%-20% or more, as measured by targeting efficiency resulting from this more granular training data and resulting model performance.
IMPACT #2: INSIGHTS FROM PREDICTIVE SEGMENTS OF POSITIVE AD RESPONDERS
Isolating predictive sub-segments of positive incremental lift provides further lift in targeting efficiency by providing insights into key audience segments and the drivers of predicted behavior by segment. Predictive segments enable a marketer to view individual positive ad response behavior at a sub-segment level for informing audience targeting and creative strategies.
These segments are not descriptive, but rather predictive and generated through a supervised model where the target variable is an incremental lift. From the enhanced training data, XAI-enabled supervised clustering allows for the grouping of individuals driven by their common, predictive weighted factors associated with their incremental lift behavior (vs descriptive features).
From an audience targeting perspective, this level of granular detail reveals “why” an individual is responding to an ad treatment and how many ad impressions or exposures to a marketing promotion it took to yield a positive response or conversion.
Predictive
Segmentation:
~ 2x Multiplier of
1:1 Targeting
Efficiency
Additionally, the degree of ad effect can be viewed if sales metrics or buy funnel steps are an input into the measurement application. These insights can be very powerful in multiplying the lift in targeting efficiency by providing a “scalpel vs. butter knife” approach to campaign analysis and planning.
Leveraging these micro-segments to target predictive lookalike audiences enables marketers to automatically identify, in rank order, prospects that look like these positive ad lift segments. Compared to an uplift model that is applied to a broader universe, these micro-segmented lookalike audiences are driven by the predictive clusters illustrated in the diagram above. This allows users to generate lookalike audiences at a much more granular level while using the explainability from the clusters to reveal the key predictive drivers for each audience, which brings us to impact #3.
IMPACT #3: CREATIVE ADJUSTMENTS BASED ON DRIVERS OF PREDICTED RESPONSE
Leveraging the insights from Beyond A/B’s predictive segments can be applied to creative adjustments – even minimal adjustments that can have a big impact. What are the key drivers of positive ad lift / conversion behavior to a treatment? If you find a segment of individuals whose key drivers of lift are based on specific spend patterns that occurred in the immediate past, or have a specific combination of demographic factors, the dimensions of creative, message and offer can be further refined to see what incremental yield these adjustments might have.
Better
Relevancy:
~ 20% Increase in
Conversion Rate
Combined with understanding optimal exposure frequency by segment and degree of effect can have can provide critical insights that further optimize conversion rates within these target groups.
THE CUMULATIVE EFFECT OF BEYOND A/B ADDS UP
The cumulative effect of increased targeting efficiency by 10%-40% or more along with an increased conversion of 20% + is a significant gain in terms of maximizing ROMI. Multiply this impact across dozens of campaigns and you can achieve significant returns by adopting the next generation of XAI enabled A/B measurement to gain new levels of performance that can only be realized through methods that enable sub-segmenting of incremental lift effect.
EVALUATING BUSINESS IMPACT
simMachines makes it easy to test through installation and backtesting of previous campaigns. Back testing can get at half the ROI, targeting efficiency specifically, but requires that sales conversion increases and yield be projected to come up with a complete picture of the potental imact of Beyond A/B.
