PREDICTING CONSUMER CONVERSION THROUGH MODEL SELECTION CRITERIA

October 01, 2020

Marketing scientists are currently enjoying an embarrassment of riches. They have an ever-growing arsenal of innovative technology tools and data streams, and they can combine them with centuries of mathematical and statistical theory and mountains of empirical research in the sciences to solve complex problems. This is truly an incredibly fun and fascinating time to work in this field.

However, because they leverage so many disparate tools and disciplines, they end up attacking similar problems in different ways. This is why friendly debates are so common in their realm. Marketing scientists constantly test the most effective approaches from one discipline against the most effective approaches from other disciplines.

A recent example of a friendly debate within our group revolves around approaches to predicting consumer conversion:

What is the better approach to predicting consumer conversion?

A) Survival analysis

B) Binary classification

C) Both

D) Neither

E) All of the above

Before I share my answer, let’s explore the two specific techniques mentioned above. Survival analysis and binary classification both have legitimate claims to be the better solution. However, they have different objectives, preconditions, and downstream uses, and they also have unique strengths and weaknesses.

Survival analysis is a branch of statistics, and when applied to marketing sciences, it centers on studying the duration of time consumers remain active before churning. This approach involves tracking marketing treatments and consumer actions throughout their life cycles and then noting the point at which they convert (purchase), no longer use your product or service, or are presumed no longer in the market. The full available dataset is analyzed over the entire data time window to understand the consumer behavior over time. Survival analysis results in high levels of transparency, interpretability, and inferential insight. It focuses on deep analysis of input features and cardinality and their relationship to conversion. It also enables integrated assessment of different scenarios and sensitivity analysis, as well as more longitudinal views of customer conversion. Its typical drawbacks, however, revolve around data availability, sample sizes, and potential computational and operational overhead.

Binary classification is an approach commonly associated with machine learning, and more specifically, supervised learning. Each customer is classified into a binary 0 or 1, depending on whether they converted. In addition, selected time periods are chosen for historical training and testing data time windows and prediction time windows. An algorithm is used to generate predictive models, and these models are continuously tuned over time to minimize prediction error.

This approach’s main objective is to provide the greatest predictive accuracy based on available training and test data, and its output is typically specific point predictions of consumer conversion within a specific time window. It does, however, generally require larger data volumes and a large amount of processing overhead to optimize the predictive accuracy. Also, unlike survival analysis, it is accompanied by transparency and interpretability concerns, and a single model typically does not enable sensitivity analysis or more longitudinal predictions of consumer conversion.

So, what’s my answer to the multiple-choice question? It’s option “E,” all of the above, because, well, it depends: The better approach to predicting consumer conversions depends on many factors that vary from company to company, team to team, and project to project.

Let’s look at an example. A CMO has budget to fund a marketing campaign with the objective of maximizing ROI (conversions over cost). The CMO wants to determine which consumers are most likely to convert on the basis of specific channels, content, images, CTAs, sequences, and timing in order to customize a campaign to target those specific audiences. If the CMO is in the early strategy and planning phases of an initiative, and if the granular data, budget, time, and marketing staff and CMO are available for a more holistic understanding, survival analysis may be the better solution.

Marketing scientists would have the ability to perform deep exploratory analyses into each input variable across channel, piece of content, image, CTA, sequence, and time-stamped interaction over an extended period. They also would be able to engage in detailed discussions with CMOs and marketing staff of exploratory outputs, inferential findings, interpretability of the models, and “what if” hypothesizing (e.g., “When are consumers most likely to convert?” “When are consumers most likely to defect?” “Why do consumers convert more frequently after receiving messaging from this channel or with this CTA?” “What combinations of channel, content, and sequence are more effective than others?”). A survival model would provide a survival curve as an output for each customer that could be very useful to answer these types of questions.

However, if the CMO has tighter budget, time, and availability constraints — but as is typically the case these days, they have abundant (somewhat noisier) data and plenty of processing power — a more specific fit-for-purpose binary classification or machine learning model that’s optimized to predict lift due to marketing communications may be better. The emphasis would be on model predictive accuracy using abundant and carefully selected training or test data, with specific outputs aligned with the needs of the selected marketing campaigns. In addition, marketing scientists could allocate their limited time a bit more on continuous improvement and tuning as they assess, retrain, and run the models on a frequent basis.

When assessing your marketing science methodology options, focus on the following criteria:

1. Deadline for delivery. Business demands typically require the shortest time to deliver a working model. This timing drives many decisions regarding how rigorous, flexible, interpretable, and accurate the model needs to be.
2. Data availability.More sophisticated, complex, and multidimensional models might require data in formats that are not readily available. Sometimes the time, effort, and complexity of wrangling data into a more specific format might not be worth it, especially when you consider the associated loss in model accuracy. Moving forward with a simpler approach with data on hand, quality checks on model predictive accuracy, and sanity checks on outputs and conclusions might be a productive and necessary decision.
3. Explainability and flexibility.On the other hand, more sophisticated and complex models often provide additional benefits, such as explainability of model outputs and flexibility to predict other outcomes. If you can gain additional explainability and flexibility benefits, it might be worth the additional upfront time and effort to identify, source, and feature engineer the data to support the model.
4. Agreeing to disagree.When faced with a choice between two solid solutions, the best decision is to pick one and agree to disagree in the short term. The marginal benefits and costs of one compared to the other might not be significant enough to prolong the battle. However, continue to monitor and measure accuracy and fit-for-purpose — business and technology conditions may shift and swing the pendulum in a different direction.

Disagreements and debates are inevitable in the marketing science world, but fortunately, they serve a great purpose. These ongoing analyses, discussions, and debates lead to enlightenment. They help us hone our skills and incrementally improve our craft. And in this specific case, they will help us provide increasingly superior consumer experiences that lead to conversions.