Probstat/notes/limit theorems

This is part of probstat.

In this section, we shall briefly describe one of the reasons why normal distributions appears very often in statistics. This also justifies our assumptions in many applications that the populations we are working on are normally distributed.

Limit theorems, important results in probability theory, consider situations when the size of the sample is very large. There are usually two types of limit results, each with many specific versions under many assumptions. They are referred to as:

• Laws of large numbers --- discuss the fact that as we increase the size of the sample, the sample mean converges to the population mean.
• Central limit theorems -- discuss the fact that the distributions of the sample mean converges to a normal distribution.

Settings: Here we describe the settings of both results. We have random variables $X_{1},X_{2},\ldots ,X_{n}$ which are independent identically distributed with mean $\mu$ and variance $\sigma ^{2}$. (This is true when we take a sample $X_{1},X_{2},\ldots ,X_{n}$ of size $n$ from a population whose mean is $\mu$ and variance $\sigma ^{2}$.)

Law of large numbers

See also the wikipedia article.

We first state the weak version of the results.

The week law of large numbers

We have random variables $X_{1},X_{2},\ldots ,X_{n}$ which are independent identically distributed with mean $\mu$ and variance $\sigma ^{2}$. For any $\epsilon >0$,

$P\left\{\left|{\frac {X_{1}+X_{2}+\cdots +X_{n}}{n}}-\mu \right|>\epsilon \right\}\rightarrow 0$

as $n\rightarrow \infty$.

It can be proved using Chebyshev's inequality.

Central limit theorems

See also the wikipedia article.

The central limit theorem

We have random variables $X_{1},X_{2},\ldots ,X_{n}$ which are independent identically distributed with mean $\mu$ and variance $\sigma ^{2}$. The distribution of ${\frac {X_{1}+X_{2}+\cdots -n\mu }{\sigma {\sqrt {n}}}}$ get closes to unit normal as $n\rightarrow \infty$. More specifically,

$P\left\{{\frac {X_{1}+X_{2}+\cdots +X_{n}-n\mu }{\sigma {\sqrt {n}}}}\leq a\right\}\rightarrow {\frac {1}{\sqrt {2\pi }}}\int _{-\infty }^{a}e^{-x^{2}/2}dx$

as $n\rightarrow \infty$.