5. Bayes' Theorem

This theorem, also known as Bayes' Law or Formula, provides a way to calculate the [posterior] probability of an event based on its [prior] (original) probability and based on occurrence of some relevant event. Let us denote the original event as A and the relevant event as B. The [prior] probability of event A is known, P(A). We want to find out the [posterior] probability of event A given event B has occurred, P(A | B). In other words, we want to know how much occurrence of event B has influenced the probability of event A to happen. Notice that events A and B occur within the sample space S. However, after we learn that event B has happened, our focus is on event A in the context of event B.

 

Bayes' Formula helps us find out the posterior probability:

Event B redefines the samples space. Now we are looking at event A with respect to event B rather than within the original sample space, S. Notice that event B can also be represented as union of two intersections of A, A^c and B (event B can occur concurrently with event A or with event A^c):

B = (A ∩ B ) ∪ (A^c ∩ B)

Since events (A ∩ B ) and (A^c ∩ B) are disjoint, the probability of their union can be expressed as the sum of their probabilities:

P(B) = P(A ∩ B ) + P(A^c ∩ B)

With a little help of the Multiplication Law we can express P(B) as:

P(B) = P(A) ⋅ P(B | A) + P(A^c) ⋅ P(B | A^c)

We can now rewrite the Bayes' Formula, by splitting event B between events A and A^c. We get an alternative form of the Bayes' Formula.

Let's verify the formulas graphically, using an example in which the sample space, S, consists of 60 identical squares. Event A is a subset of the space (blue squares). It is made of 25 squares. Obviously the complement of event A, A^c, contains 35 green squares (60 - 25). Event B occurs as a set of 16 orange squares. What is the probability of event A given event B?

Based on the sizes of the events (sets), we can find out the probabilist: P(A) = 25/60, P(B) = 16/60. Furthermore, if event A (consisting of 25 squares) has occurred, only 4 squares that belong to A also satisfy event B. Thus P(B | A) = 4/25.

In order to verify the alternative formula we also need the two other probabilities: P(A^c) and P(B | A^c). Since A^c is a complement of A, then P(A^c) = 1 - P(A) = 1 - 25/60 = 35/60. This result is consistent with the number of squares (35), contributing to event A^c out of the total of the squares (60) in the sample space (S). If event A^c occurs, then 12 its squares will satisfy event B. Thus P(B | A^c) = 12/35.With all the continuance elements of the alternative formula, we get:

Finally, this problem, P(A | B), can also be solved by analyzing events A and B with respect to the Multiplication Law.

There are 4 squares, satisfying event A that belong to the new sample space (B). (They are a product of the intersection of events A and B.) Since event B consists of 16 squares, P(A ∩ B) = 4/16 = 1/4.

In conclusion, this result, P(A | B) = 1/4 = 3/12, should not be surprising. The prior probability, P(A) = 25/60 = 5/12. However, when event B (a new fact) has occurred, it contributed more to the complement of A (A^c) than to A. Thus, the posterior probability for event A has decreased (from 5/12 to 3/12).

1. What is the probability of number 5 (A = {5}) to come up.

Using the Classical definition, since there is only 1 event in favor of {5} out of 6 possible events, P(A) = 1/6.

Since the events A = {1, 3, 5} and B = {2, 4, 6} are disjoint, their intersection is an impossible event (∅). Thus P(A ∩ B ) = P(∅) = 0.
Except for being equally likely (1/2), events A and B have nothing in common. If one occurs the other does not.

Moreover, these events are complementary since they are disjoint and their union constitutes the sample space.

The following relations are true about the events:

3. Given one of numbers 2, 3, or 4 has been selected (event B = {2, 3, 4} has occurred), what is the probability that it was an odd number.

Here the sample space shrinks to event B. In this space, consisting of 3 elementary events, there is only one odd number ({3}). Thus:

Now, consider an experiment with coins and marbles in jars. A coin (1 Penny) is flipped twice. Such a flip produces an event, consisting of pairs of heads (H) and/or tails (T). Jar A contains 4 Red and 3 Blue marbles. Jar B contains 2 Red and 5 Blue marbles. Assume that our experiment has two phases. In Phase 1, the coin is flipped randomly (independently) twice. Phase 1 returns one of the following, equally likely, events: {HH, HT, TH, TT}. In Phase 2, a marble is selected from one of the jars, depending on the outcome of Phase 1. We are interested only on two possible outcomes of Phase 1: {HH} or {HT, TH, TT}. Notice that the latter can also be expressed as a complement of the former, {HT, TH, TT} = {HT}^c. If Phase 1 returns {HH} than a marble is selected from Jar A, otherwise it is selected from Jar B. The following image depicts the experiment structure and its probabilities.

Since in Phase 1, the four possible events are equally likely, then the probability of getting two Heads is 1/4, P({HH}) = 1/4. Notice that this probability can also be calculated as P({H}) ⋅ P({H}) = (1/2)(1/2) = 1/4. The probability of event {H} ∩ {H} is the product of the probabilities since the outcomes of the individual events are produced by independent flips of the coin. (Recall that, for independent events, A and B, we have: P(A ∩ B} = P(A) ⋅ P(B).)
The probabilities of selecting marbles from the jars depend on which jar the marbles are selected from. There are two pathways for selecting a Red marble or Blue marble. The end of each pathway represents a joint event of Phase 1 and Phase 2 outcomes (events), resulting in a Red or Blue marble. Using the Multiplication Law, the probabilities are assessed as follows:

From the decision tree, we learn that there are two alternative and disjoint ways of getting a Red marble selected: one via Jar A and the other via Jar B:

The previous diagram shows the probabilities associated with these alternatives. Thus:

5. What is the probability of selecting a Blue marble in Phase 2 given Phase 1 resulted in HT, TH, or TT?

Event {HT,TH,TT} is equivalent to {HH}^c. Therefore, if this event has occurred, then automatically Jar B has been selected. There are 5 Blue marbles in jar B out of the total of 7 marbles. Thus:

Perform a similar analysis for P({Jar B} | {Red}).

Recall that the a priori probability of selecting Jar B is 3/4. This is the probability of not getting 2 Heads in Phase 1, P({Jar B}) = P({HH}^c) . With additional knowledge of the fact that a Red marble has been selected, the posteriori probability, P({Jar B} | {Red}), goes down to 3/5. A contributing factor for this decreased probability is the smaller population of Red marbles in Jar B.

Baseball Hockey Total Woman 16 6 22 Man 6 12 18 Total 22 18 40

Baseball Hockey Total Woman 0.40 0.15 0.55 Man 0.15 0.30 0.45 Total 0.55 0.45 1

Events A^c={MPG<20}, B^c={MPG>30} are disjoint. Thus their intersection is empty,
A^c ∩ B^c = ∅

Also in this case, one can apply the De Morgan's law, A^c ∩ B^c = (A ∪ B)^c.
Since union A ∪ B = S = {-∞ < MPG < +∞}, its complement is impossible
(A ∪ B)^c=∅

a. Based on the above information, what are the probabilities that can be established?

b. Are the events A and B mutually exclusive and/or exhaustive? Explain.

c. Are the events A and B independent? Explain.

d. What is the probability of selecting an American parent given that she/he is a spying parent? We want to calculate P(A | B).

# The probabity of event A.

pA = 0.10

# The probabity of event B.

pB = 0.44

# The probabity of event B given A, P(B | A).

pBgivenA = 0.60

# The probability of joint events A and B.

pAandB = pBgivenA * pA
pAandB

[1] 0.06

# A condition for independent evants.

pBgivenA == pB

[1] FALSE

# The probability of A given B, P(A | B).

pAgivenB = pAandB / pB 
pAgivenB

[1] 0.1363636

An alternative solution, using a contingency table. We can built such a table by cross-referencing the probabilities of events A, A^c vs. B , B^c.

We start with what we know.

	B	Bc	Total
A	0.06		0.10
Ac
Total	0.44

Notice that 0.06 is the probabilty of joint events A and B, P(A and B), which is equal to P(B | A) * P(A). Next, we can fill the probabilities of the complementary events: P(A^c) = 1 - P(A) and P(B^c) = 1 - P(B). The grand total (1.00) comes from the mutually exclusive and exhaustive events A and A^c as well as B and B^c, P(A or A^c) = P(A) + P(A^c) = 1, or P(B or B^c) = P(B) + P(B^c) = 1.

	B	Bc	Total
A	0.06		0.10
Ac			0.90
Total	0.44	0.56	1.00

The rest is easy. The colum Total and row Total are marginsl probabilities. They are sums of the related joint probabilities in rows and columns.
Row 1:
P(A) = 0.10 = P(A and (B or B^c) ) = P(A and B or A and B^c) = P(A and B) + P(A and B^c) = 0.06 + P(A and B^c) = 0.10.
Thus P(A and B^c) = 0.10 - 0.06 = 0.04.
Column 1:
P(B) = 0.44 = P(B and (A or A^c) ) = P(B and A or B and A^c) = P(B and A) + P(B and A^c) = 0.06 + P(B and A^c) = 0.44.
Thus P(B and A^c) = 0.44 - 0.06 = 0.38.
We can now add the probabilities to the table.

	B	Bc	Total
A	0.06	0.04	0.10
Ac	0.38		0.90
Total	0.44	0.56	1.00

In a similar way we can complete the missing entry.
Row 2:
P(A^c) = 0.90 = P(A^cand (B or B^c) ) = P(A^c and B orA^c and B^c) = P(A^cand B) + P(A^cand B^c) = 0.38 + P(A^cand B^c) = 0.90.
Thus P(A^cand B^c) = 0.90 - 0.38 = 0.42.
Here is our complete contingency table.

	B	Bc	Total
A	0.06	0.04	0.10
Ac	0.38	0.42	0.90
Total	0.44	0.56	1.00

We can now answer the original questions.
b. Event A and B are not mutually exclusive since their joint pobability is not equal to zero, P(A and B) = 0.06. They are also not mutually exhaustive. If they were, the probability of their union would have to be one.
We have P(A or B) = P(A) + P(B) - P(A and B) = 0.10 + 0.44 - 0.06 = 0.48.
c. Events A and B are not independent. If they were independent then the probability of any of the events would have to be equal to the conditional probability with respect to the other one. A shorter way to show it is by checking if that the probability of the joint events is not the same as the product of their probabilitities:
P(A and B) = 0.06 ≠ P(A)*P(B) = 0.10 * 0.44 = 0.044.
d. What is P(A | B)? Assuming that event B has happend, what is the probability of event A to occur? If we know that event has happened, then the entire probability space got reduced to column B. Thus this conditional probability is equal to 0.06/0.44 = 0.1363636. This comes from formula P(A or B) /P(B). Recall that the Multiplication Law states:
P(A or B) = P(A | B)*P(B) which is the same as P(B | A)*P(A) .

a. What is the probability that a randomly selected patient experienced cardiac arrest during the graveyard shift?

b. What is the probability that a randomly selected patient survived for discharge?

c. Given that a randomly selected patient experienced cardiac arrest during the graveyard shift, what is the probability the patient survived for discharge?

d. Given that a randomly selected patient survived for discharge, what is the probability the patient experienced cardiac arrest during the graveyard shift?

e. Are the events "Survived for Discharge" (Yes) and "Graveyard Shift" (Graveyard) independent? Explain using probabilities. Given your answer, what type of recommendations might you give to hospitals?

# Set up a data frame ca (cardiac arrest).

ca = read.table(text="Yes No
11604 46989
4139  24016", header = TRUE)
ca

    Yes    No
1 11604 46989
2  4139 24016

# Add column Total.

ca$Total = rowSums(ca)
ca

    Yes    No Total
1 11604 46989 58593
2  4139 24016 28155

# Add row Total.

ca = rbind(ca, colSums(ca))
ca

    Yes    No Total
1 11604 46989 58593
2  4139 24016 28155
3 15743 71005 86748

# Name rows.

row.names(ca) = c("DayEvening","Graveyard","Total")
ca

             Yes    No Total
DayEvening 11604 46989 58593
Graveyard   4139 24016 28155
Total      15743 71005 86748

a. Solution. What is the marginal probability for event Graveyard, P(Graveyard)?

ca["Graveyard","Total"]/ca["Total","Total"]

[1] 0.3245608

b. Solution. What is the marginal probability for event Yes, P(Yes)?

ca["Total","Yes"]/ca["Total","Total"]

[1] 0.1814797

c. Solution. What is the conditional probability, P(Yes | Graveyard)?

ca["Graveyard","Yes"]/ca["Graveyard","Total"]

[1] 0.1470076

d. Solution. What is the conditional probability, P(Graveyard | Yes)?

ca["Graveyard","Yes"]/ca["Total","Yes"]

[1] 0.2629105

e. Solution. Are events "Graveyard", "Yes" independent? Is P(Graveyard|Yes) = P(Graveyard)?

ca["Graveyard","Yes"]/ca["Total","Yes"] == ca["Graveyard","Total"]/ca["Total","Total"]

[1] FALSE