The Deal with the Denominator

fieldnotes
3 min readJul 25, 2021
A can of soda will have have some volume of soda to consume

The denominator of Bayes Rule is arguably the most important part of the formula.

Motivation:

Why?

Let’s go through the formula piece by piece for a second:

Posterior: A proper probability distribution.

Likelihood: Usually not a proper probability distribution.

Prior: Often a proper probability distribution.

Denominator: a probability distribution.

Which of these is unlike the others?

The likelihood!

If the likelihood is not a probability distribution, then we can assume that the product of the likelihood and prior isn’t either. So then, how does the numerator become the proper posterior?

That work is done by the work of the denominator, or p(data), which represents the beliefs held for ALL POSSIBLE DATA samples that can be collected.

Once we have data that is collected, the denominator probability distribution simplifies to some numeric value. This value “scales” the numerator of Bayes Rule so that it can transform from NOT a proper probability distribution to a proper one.

Image Cred: fieldnotes

Soda Volume Analogy:

Let’s keep thinking.

Consider a situation where you are responsible for inspecting soda can soda volume.

In terms of soda volumes, the numerator of our hypothetical Bayes Rule calculation represents the data collected from soda volumes of recent batches. The denominator represents the TOTAL soda volume that is expected in an ordinary day.

Image Cred: fieldnotes

So the denominator acts like a baseline foundation or context for the incoming data that colors the numerator.

Calculating the Denominator of Bayes Rule:

So how to we calculate this denominator?

You may recall the denominator of Bayes Rule doesn’t depend on the parameters of choice — ONLY data. So how do we get rid of this parameter dependence that is clearly part of both the prior and likelihood?

Image Cred: fieldnotes

We sum the numerator over all parameters (integrate for continuous cases) to obtain the denominator.

Image Cred: fieldnotes

Looks complicated doesn’t it? Turns out, it is complicated in practice. However, we have a lot to cover before we dive deep into the complexities, so hang tight!

Main Takeaway: The denominator of Bayes Rule is what makes the posterior a proper probability distribution. It also serves to provide a bird’s eye view of what kind of data we are expected to collect in the future, given some model to start with.

Research done from: “The devil is in the denominator” A Student’s Guide to Bayesian Statistics, by Ben Lambert, SAGE, 2018.

--

--