lacunary - Mathnotes

Probability Basics

Sample Space

Definition: Sample Space \@{sample-space}

The set of all possible outcomes of a statistical experiment is called the sample space and is represented by the symbol S. Each outcome in a sample space is called an element or member of the sample space or simply a sample point.

A sample space must be non-empty.

Referenced by (9 direct, 145 transitive)

Direct references:

Transitive (depth 1):

Transitive (depth 2):

Transitive (depth 3):

For example, the sample space representing the outcomes of the rolls of a standard six-sided die is

S={1,2,3,4,5,6}.

Definition: Event \@{event}

An event is a subset of a sample space. For example, we can say the event A that the roll of a die is odd is A={1,3,5}, the event B that the roll of a die is even is B={2,4,6}, and that the event that the roll of a die is less than or equal to 4 is C={1,2,3,4}.

Referenced by (10 direct, 123 transitive)

Direct references:

Transitive (depth 1):

Transitive (depth 2):

Transitive (depth 3):

Transitive (depth 4):

Transitive (depth 5):

A lot of the rest of basic probability is just set theory from there:

  • The event that the roll of a die is not less than or equal to 4 is the complement of C, written as Cc={5,6}.
  • The event that the roll is both even and less than or equal to 4 is the intersection of A and C, written as A∩C={2,4}.
  • The event that the roll is either even or less than or equal to 4 is the union of A and C, written as A∪C={1,2,3,4,6}.

We can also have continuous sample spaces. For example, if we were to randomly choose a number between 0 and 1 inclusive, the sample space would be S=[0,1], and the event that the outcome is a number whose decimal digits are only 2s would be {0.2,0.22,0.222,…}.

Probability

Definition: Probability \@{probability}

Note: This applies to sample spaces of discrete events and is a bit hand wavy.

We can assign a probability or weight to each sample point in a sample space by giving it a value ranging from 0 to 1. Events that are more likely to occur have a probability closer to 1, and events that are less likely to occur have a probability closer to 0. We give a probability to all sample points in a sample space such that the sum of the probabilities of all sample points in a sample space is 1.

Then, the probability of an event A is the sum of the probabilities of all sample points in A. Therefore,

0≤P(A)≤1,P(∅)=0,P(S)=1.

Referenced by (33 direct, 96 transitive)

Direct references:

Transitive (depth 1):

Transitive (depth 2):

Transitive (depth 3):

Transitive (depth 4):

Definition: Odds Notation: odds \@{odds}

The odds of an event A occurring are defined as

odds(A)=P(A)P(Ac)=P(A)1−P(A)

that is, the probability A occurs divided by the probability of the complement of A (i.e., the probability that A does not occur.)

Referenced by (3 direct)

Direct references:

Note \@{note-5}

Odds are often written as a @ratio of positive integers. So we'd say that the odds of drawing an ace from a standard deck of 52 cards is

odds(Ace)=4524852=448=112=1:12.

Definition: Log Probability \@{log-probability}

A log probability is just a logarithm of a probability.

Referenced by (2 direct, 46 transitive)

Direct references:

Transitive (depth 1):

Transitive (depth 2):

Transitive (depth 3):

Transitive (depth 4):

Note \@{note-7}

Using log probability has some advantages:

  • It converts multiplication of probabilities into addition, which are faster to compute.
  • It improves @numerical stability when very small probabilities are involved.
  • It eliminates @exponentiation.
  • Logarithms are concave which is useful for @optimization.

Probability Rules

Theorem: Addition and Multiplication Rules \@{probability-rules}

The probability of A or B is the probability of A plus the probability of B minus the probability of A and B occurring together:

P(A∪B)=P(A or B)=P(A)+P(B)−P(A and B)

If A and B are mutually exclusive events, P(A∪B)=P(A)+P(B).

We have to subtract the overlap between A and B to avoid double counting.

Similarly

P(A∩B)=P(A)+P(B)−P(A∪B).

The probability of A and B is the probability of A times the probability of B given A, or equivalently, the probability of B times the probability of A given B.

P(A∩B)=P(A and B)=P(B and A)=P(A)⋅P(B|A)=P(B)⋅P(A|B)

If A and B are independent, this reduces to P(A)⋅P(B).

Definition: Conditional Probability \@{conditional-probability}

From this, we can give the formula for conditional probability. The probability of A given B is

P(A|B)=P(A and B)P(B).

That is to say, the probability of A occurring given B has occurred is the portion of times B occurs that A also occurs.

If P(A)=P(A|B), then A and B are independent events, and P(A)⋅P(B)=P(A and B).

Referenced by (3 direct, 6 transitive)

Direct references:

Transitive (depth 1):

Transitive (depth 2):

Theorem: Bayes' Theorem \@{bayes-theorem}

TODO: state and prove this.

Referenced by (2 direct)

Direct references:

TODO: independence, conditional probability, and conditionally independent are stated here for events. The random variable versions are missing, and pages that reason about random variables link to these blocks for lack of them.