Please don’t click away if you prefer “Bayes Law” or “Bayes Formula”.
The point of this post is to use the foundations of probability theory to prove the foundation of Bayesian statistics — Bayes Rule.
Here is a list of basic properties we will need. We are assuming A and B are two events.
Bayes Rule can be derived from this following relation, which will be explored in the next post:
The above relation reads as “Probability of Event A occurring , given Event B occurred equals the Probability of Event B occurring, given A occurred”.
At this point, I had two questions. 1) How did the result above come to be? and 2) Does it matter if A and B are (in)dependent?
2) can be answered by the following proofs:
2a) Assume A and B are dependent
2b) Assume A and B are independent
The answer to 2) is no.
This is due to the Commutative Intersection Property listed above, which answers 1). Both proofs use this property as a hinge (you can check!).
These are just a couple proofs. It is possible to prove Bayes Rule without using Joint Probability as well.
I never actually mention Bayes Rule explicitly anywhere in the post, so here is the straightforward proof after having the knowledge and satisfaction of proving P(A,B) = P(B,A):
Try it out and divide by P(A) this time.
What you observe is the power of Bayes Rule, which is to convert P(A|B) to its inverse P(B|A). We will discuss this further in a later post.