I often get asked for a POV on this, so I guess a lot of people are asking the same question.

Recap: What are Priors?

In a Bayesian media mix model, priors are simply what you believe the results might be before you look at the data or do any modelling. Think of them as expressing real-world knowledge — like “it’s very unlikely that my sales go down when I spend more on advertising” — in mathematical language. [5].

My view, in summary: Priors introduce risks of bias which can undermine model integrity — particularly when the priors are derived from last‑touch, platform‑reported metrics or poorly specified experiments. The core issues are that last touch platform reporting is biased to last touch and therefore inaccurate, and experiments are rarely undertaken with the levels of statistical rigour required to make them reliable.

Potential Advantages of Using Priors

Let’s look at the potential advantages of using priors.

1. The biggest argument for Bayesian Priors is that they reduce uncertainty in an MMM environment. Bayesian MMM doesn’t just give you a single point estimate for the impact of each marketing channel. It provides a probability distribution of possible impacts… This allows you to understand the uncertainty associated with the estimates, leading to more informed decision-making [6].
2. Model outputs (e.g. CPA by channel) look “right” and are relatable in the real world: To a large extent, priors fix the model outcomes into sets of values that seem to match the marketing team’s expectations and experience.
3. Priors can help when the data available for modelling is limited: Sometimes there isn’t sufficient spend data to accurate estimate coefficients for media channel performance. Spend may be too low, or the spend in one channel is drowned out by higher spends in other channels at the same time, or the number of weeks with spend may be very low. In these situations, priors can help by preventing coefficients from collapsing toward zero because the data is weak.
4. Priors help when spend variables are highly correlated: Many media campaigns feature multiple channels running at the same time campaigns often take place over 2–8-week periods with media channels being combined to extend reach or delivery efficiency frequency. This means input variables are highly correlated with each other which is a problem for regression models – they can’t isolate the effects of correlated channels. Using priors in a model helps the model find clarity by providing guides to what channels effects should be.

Key Risks and Limitations

1. Priors are beliefs, not data, so they risk injecting bias into your model: A key issue is that Priors are beliefs about media performance. Because they are beliefs about media performance e.g. “our Last Touch Social CPA is always between £10 and £20, so let’s fix that range into our MMM results” – we risk using those findings to inform the model outcomes – clearly this is a form of confirmation bias [7].
2. Priors often rely on inaccurate last‑touch or platform‑reported ROI Data: In practice, most priors come from platform dashboards rather than controlled experiments. The problem with platforms is that they generally provide last touch attribution (sometimes, first etc, but never full funnel i.e. including trend and seasonality an marketing mix variables for example).  This means that from the “get-go” Last Touch reporting is inaccurate – our industry knows that- so why use these results as priors in MMM?  But obviously using Last Touch platform sourced priors is only going to transfer that uncertainty directly into your marketing mix model. This is ironic because most MMMs are commission to provide an alternative view to Last Touch reporting.
3. Priors can pull MMM results toward last‑touch, and away from more statistically reliable findings: One of the big arguments for using priors is that they keep MMM results grounded in reality. But if you are using priors, you are creating an artificial result. If the prior is biased (e.g. towards last touch reporting), the model output will also be biased. Whilst this can make the MMM appear more aligned with platform reporting, but that alignment is artificial and the results are erroneous.
4. Priors can destabilise other coefficients: Let’s go back to high school maths. We know that equations have to balance on both sides – output e.g. Sales on the left, inputs e.g. media spend on the right. If we fix one or more of the input components of our equation on the right, other parts of it will have to move in order to accommodate that prior fix. This means your model will almost certainly inflate some media channels, suppress other, channels especially upper‑funnel media and produce inaccurate results
5. Priors reduce transparency: Stakeholders often struggle to understand how much of a coefficient is “data‑driven” versus “prior‑driven,” which can undermine trust.
6. Priors can mask genuine performance shifts: If a channel’s true effectiveness changes (e.g., due to creative fatigue, privacy changes, or market dynamics), a strong prior can prevent the model from detecting it. Equally performance might improve – dramatically. Let’s say you launch a new social media campaign with a new offer and new creative. Your MMM Social CPA falls from between £10 and £20 to £5. That means your social campaign has become much more effective. But MMM priors would likely exclude that result – or at least suggest it is unrealistic.
7. Priors Do Not Include Adstock or Diminishing Returns: There is a common belief that priors somehow incorporate adstock or saturation assumptions. They do not. Adstock and diminishing returns are structural modelling choices — they define how media works overtime and at different spend levels. They are often subjectively judged but they can and should be extracted from the MMM dataset itself using a grid search loss minimisation technique [8].   This can’t be circumnavigated by experiments as Adstock and diminishing returns are critical parts of advertising evaluation [9]. They can only be determined from  long time-series datasets, not short‑term experiments.
8. Priors can come from experiments, but these are notoriously difficult to get right: Priors can be defensible when grounded in rigorous, repeatable experiments. However, this is a technically demanding area. Getting marketing experiements right requires:
   1. Clean treatment and control regions
   2. No spillover
   3. Stable delivery
   4. Repeatability
   5. Sufficient spend, time and statistical power

In practice, achieving this level of scientific discipline in marketing experiments is difficult. Geographical regions can be more fluid than they look on a map – we live in a mobile world when people can move from one region to another in very short periods of time. [10].

Real‑World Example: Identical Creative, Different Results

There have been cases where a team ran an A/B test using the same creative in both treatment and control. Despite being identical, the experiment produced different lift results for each “creative.”  This highlights how algorithmic targeting, user heterogeneity, and data aggregation conspire to confound the magnitude, and even the sign, of ad A/B test result [11]. If identical creatives can produce different results, it shows how fragile and noisy marketing experiments can be — and why they often lack the stability required to serve as robust priors.

Conclusion:

I conclude with these points:

1. Avoid using Bayesian priors if you want a high integrity MMM.
2. Not using priors may produce more challenging results – but isn’t that what you want? You are building your MMM to get a different perspective on your marketing performance; to identify hidden opporutunities and to improve marketing ROIs. Why gloss over that valuable insight?
3. This may result in a more challenging conversation with the C-suite – but it’s safter to report a challenging result from a high integrity model than any result from a flawed model.
4. If you use priors to shape the answer, then the answer will look better. That doesn’t mean it’s the right answer, in as much as any model can produce the right answer.
5. Priors introduce confirmation bias into your model. If you believe the enemy of good modelling is bias, you must question the use of priors in your MMM project. They may be acceptable to some, but not to others.
6. If you are going to use priors, you must be very careful about where you source them. There are three main sources – Experience, Platforms and Experiments, but all have flaws and all run the risk of introducing bias into your model.

Implications for Marketers – checklist

• If you are going to use priors you must be certain that they are accurate.
• Be aware that in most marketing contexts the risks of priors being inaccurate are high.
• Don’t rely on last touch data for priors.
• If you use experiments, ensure they are properly specified, and even then, use them with care.
• In cases where you have too little data for full MMM, be careful about using Bayesian priors to overcome this problem
• In data light situations, consider reviewing your data, accepting lower granularity or looking at a different attribution modelling technique like regularisation.

References

1. J Martin and P Perez, Frequentists vs Bayesians and Marketing Science, Quantified Nation, July 2024 – https://open.substack.com/pub/quantifiednation/p/qn9-frequentists-vs-bayesians-and
2. Hits and Misses of Meridian – A Thorough Deep Dive, Aryma Labs Feb 2025 https://arymalabs.substack.com/p/hits-and-misses-of-meridian-a-thorough
3. Duncan Stoddard, Is Bayesian MMM worth the faff? DS Analytics Blog, February 2024, https://dsanalytics.co.uk/thoughts/is-bayesian-mmm-worth-the-faff
4. Two key problems that ail Bayesian MMM – Aryma Labs April 2024 https://arymalabs.substack.com/p/two-key-problems-that-ails-bayesian
5. Marty Sanchez, What Are Priors in MMM – And Why They’re Difficult to Get Right (But You Need To) Get Recast Blog June 2025 – https://getrecast.com/what-are-priors-in-mmm-and-why-theyre-difficult-to-get-right-but-you-need-to/
6. Rohit Nair, What is Bayesian MMM & why use it? Medium April 2025 – https://medium.com/@rohitnair.inft/what-is-bayesian-mmm-why-use-it-942d0193e7eb
7. Cognitive bias and data: how human psychology impacts data interpretation – Penn LLPS Features October 2025 https://lpsonline.sas.upenn.edu/features/cognitive-bias-and-data-how-human-psychology-impacts-data-interpretation
8. Nephade, D., A Predictive Modeling Approach to Multi Objective Marketing Mix Optimization: Balancing Performance, Acquisition, and Efficiency – in International Journal on Science and Technology (IJSAT) Volume 16, Issue 1, January-March 2025.
9. Gijsenberg et al., Understanding the Role of Adstock in Advertising Decisions, SSRN, 2011.
10. Tyler Buffington, Eppo, The Bet test – Spotting Problems in Bayesian A/B Test Analysis Dec 2024, https://www.geteppo.com/blog/the-bet-test-problems-in-bayesian-ab-test-analysis
11. Braun and Schwartz, Where A/B Testing Goes Wrong: How Divergent Delivery Affects What Online Experiments Cannot (and Can) Tell You About How Customers Respond to Advertising, Journal of Marketing, American Marketing Association, August 2024. https://journals.sagepub.com/doi/abs/10.1177/00222429241275886

The post Using Bayesian Priors in MMM: Do Better‑Looking Results Mean You Have a Better Model? first appeared on Marketing IQ.

A marketing Mix Model (MMM) can produce challenging results, especially if you have been relying on last touch( LT) attribution.  When presented with challenging results your first question may well be “can I trust the model?”.

The answer to this depends largely on the quality of the model as expressed through a set of diagnostics. The problem is not all diagnostics are shared.

Let’s explore MMM diagnostics in some more detail so you are better placed to form your own conclusions on model quality.

What you should get from your MMM partner

Let’s start with the diagnostics you should see from your MMM partner. They should present you with these test diagnostics:

1. Exploratory Data Analysis (EDA)

• Correlation Analysis (before modelling): Identifies highly correlated variables early, helping avoid multicollinearity in the model build and guiding variable selection – this is important because correlated variables can destabilise any modelling and results.
• Trend & Seasonality Checks: These checks confirm long‑term trends and seasonal peaks are properly captured before modelling media effects – this is important because if you don’t isolate these components, you risk incorrectly misattributing their effects to media.
• ACF (Autocorrelation Function): Ensures time‑based patterns are understood and removed so the model isn’t biased by serial correlation – this is also important. Your sales or revenue data can include patterns or “tidal effects” where your sales in the current weeks are partly related to sales in the previous week. Again, if you don’t control these relationships, your model will produce erroneous results.

2. Model Fit & Accuracy

• R²: This is a diagnostic that most marketers will be familiar with – it shows how much of the sales movement the model can account for by the model – but beware this is by no means a full diagnostic of the model.
• Adjusted R²: A fairer version of R² that penalises unnecessary variables, but again, not a full diagnostic.
• Actual vs Fitted plot which shows how well the model’s estimates match the actual data.
• MAE (Mean Absolute Error): The average size of the model’s prediction error in real sales units. For example, if a typical week delivers 1,000 sales and the model predicts values between 900 and 1,100, the absolute error is 100 units.
• MAPE (Mean Absolute Percentage Error): The model’s average prediction error expressed as a percentage of actual sales. Using the example above, an absolute error of 100 on 1,000 sales corresponds to a MAPE of 10%.

3. Residual Diagnostics

• Durbin–Watson test: Checks whether any time‑based pattern is left in the errors; values near 2 indicate no remaining autocorrelation
• Breusch–Pagan test: This test checks whether error variance is stable; instability can affect model coefficient reliability

4. Model Stability & Interpretability

• VIF (Variance Inflation Factor): Helps you measure the correlation between channels in a model and helps ensure channels aren’t too correlated to separate cleanly. If your model contains variables that are highly correlated with each other, it is not likely to be reliable.

These tests are all mission critical for evaluating both the data going into the model and the  model itself. But while they diagnose the quality of the model, they do not evaluate its predictive quality.  Predictive quality tests are further tests that assess the model’s ability to predict outcomes on previously unseen test data. In some ways these validation tests are the  ‘acid test’ in model diagnostics. When you combine the standard diagnostics (1-4 above) with the predictive quality tests you will see:

1. Good results in both mean you are dealing with a reliable model.
2. Good results in one but not the other should be cause for concern.
3. Poor results in both diagnostics and validation tests mean the model should be ignored and re-specified.

Unfortunately, the predictive quality tests are not always shared by MMM providers – I’ll leave you to figure out why.

So, what are the predictive quality tests?

MMM validation route 1: The holdout test

• How does the test work? In this test, we test the model on a subset of the data, usually towards the end of the sample. Let’s assume you have a model built on three years’ worth of weekly data – that’s 156 observations. You can ‘slice off’ the last 26 weeks and use the first 130 observations to “train” the model. You can then apply the trained model to the last 26 weeks, also known as the “test” period.
• What should you see?  You will have the actual sales data for the 26 week test period. The key ‘acid test’ question for you is how well the model’s prediction for those 26 test  weeks match actual sales in the test period. If you model is predicting within +/- 20% that’s a strong result. If it’s predicting between +/- 10% that is a very strong result.

MMM validation route 2: Cross Fold validation

• How does this test work? In this test we slice your 156 weeks of data into 10 consecutive time “blocks” of 15-16 weeks. In a similar way to the holdout test, we provide the explanatory variables for each block  and ask the model to predict what it estimates sales to be in each of these blocks based on the model coefficients. We then examine the prediction vs the actual and the size of the error in each block.
• What should you see?  We will have the actual sales data for each of the blocks. Again, the key ‘acid test’ question for you is how well the model’s prediction for those blocks matches actual sales in each of those blocks. If your model is predicting within +/- 20% that’s a strong result. If it’s predicting between +/- 10% that is a very strong result.  It’s worth noting that in some categories the 20% might be too tight, you could relax it to say 20-30%, but you need to be close to that range to have a validated model.

What do these results mean? If you see a model prediction within 20% of the actual in this holdout period, then you have a model that is working well. It is unlikely to be ‘overfitted’. Overfitting is a way of making the model look good within the sample, but it  does not perform well outside the sample and would therefore fail these tests.

The post How reliable is your marketing mix model (MMM)? first appeared on Marketing IQ.

Establishing incrementality is critical if you want genuine brand growth. Why? Because many performance platforms collect, report and even double-count sales from multiple sources, including those which might happen even if you didn’t run any activity. This means you are attributing to media spend sales that would have happened without media spend. This type of misattribution will mean you are using flawed data for budget optimisation and this in turn will lead to sub-optimal media performance. Misattribution makes your budget less efficient and less effective.

In order to detect incrementality we need to establish what would happen if your product or service didn’t have any marketing activity. There are three things – sometimes called “components” to look at here:

1. Trend – what is the underlying trend in your category an din your sales – are sales they in growth, decline or stable?
2. Seasonal cycles – What are the repeating patterns in the data – do sales increase or decrease in certain months, certain weeks on a regular predictable pattern?
3. Base brand equity – how many sales would you expect to see if you paused your marketing activity

These components can often account for more than 75% of your sales revenue. If your performance platforms are reporting 100 sales, it could be the case that 75 of these sales would have happened without any marketing activity. For many advertisers this is an “OMG” moment.

Imagine if you could identify the sales that would have happened without marketing or media support and then focus your marketing budget on activities that deliver genuine incremental growth rather than paying a platform “tax” for sales that were going to progress through your sales pipeline without any short-term marketing spend.

Let’s take a closer look at trend and seasonality and why it’s important. We’re going to use the “Bike Sales” dataset from Kaggle.

First let’s look at the sales data itself:

Here we can see bike sales from July 2017 to July 2022 over a total of 260 weeks.  We can make some initial observations. There is an underlying growth trend. We can also see that there are a number of peaks and troughs in the data. We see that the highest sales weeks are around 110k and the lowest sales weeks are around -30k so the weekly sales have a range of c. 140k.

Now let’s extract the trend component from the dataset:

We can see the underlying trend in the data, quantified using a moving average. We can see there is a strong upward trend from 50k sales to almost 85k sales.

Next, let’s extract the seasonality component from the data set:

It’s important to note here that “seasonality” doesn’t mean “seasons” as in Spring, Summer, Autumn and Winter. Here seasonality refers to any repeating cycles in the data. We can see there is  clear pattern of repeating cycles. These repeating cycles range from +10k to -20k.

And finally we are left with the Random component:

The Random component represents sales that are not explained by trend and seasonality. You can see that these random sales i.e. not explained by trend or seasonality, range from about +35k to -30k.

This random data is the data we test for contributions from media spend.  More on that model and its outputs in the next post.

The post What is incrementality in marketing – extracting trend, seasonality and brand equity first appeared on Marketing IQ.