Probstat/notes/sample means and sample variances

This is part of probstat

Consider a certain distribution. The mean $\mu$ of the distribution is the expected value of a random variable $X$ sample from the distribution. I.e.,

$\mu =E[X]$.

Also recall that the variance of the distribution is

$\sigma ^{2}=Var(X)=E[(X-\mu )^{2}]=E[X^{2}]-E[X]^{2}.$

And finally, the standard deviation is $\sigma ={\sqrt {Var(X)}}$.

Sample Statistics

Suppose that you take $n$ samples $X_{1},X_{2},\ldots ,X_{n}$ independently from this distribution. (Note that $X_{1},X_{2},\ldots ,X_{n}$ are random variables.)

Sample means

The statistic

${\bar {X}}={\frac {X_{1}+X_{2}+\cdots +X_{n}}{n}}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}$

is called a sample mean. Since $X_{1},X_{2},\dots ,X_{n}$ are random variables, the mean ${\bar {X}}$ is also a random variable.

We hope that ${\bar {X}}$ approximates $\mu$ well. We can compute:

$E[{\bar {X}}]=E\left[{\frac {1}{n}}\sum _{i=1}^{n}X_{i}\right]={\frac {1}{n}}E\left[\sum _{i=1}^{n}X_{i}\right]={\frac {1}{n}}\sum _{i=1}^{n}E[X_{i}]={\frac {1}{n}}n\mu =\mu$

and since $X_{1},X_{2},\ldots ,X_{n}$ are independent, we have that

$Var({\bar {X}})=Var\left({\frac {X_{1}+X_{2}+\cdots +X_{n}}{n}}\right)={\frac {1}{n^{2}}}Var(X_{1}+X_{2}+\cdots +X_{n})={\frac {1}{n^{2}}}\cdot n\cdot Var(X)={\frac {\sigma ^{2}}{n}}.$

Sample variances and sample standard deviations

We can also use the sample to estimate $\sigma ^{2}$.

The statistic

$S^{2}={\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{n-1}}$

is called a sample variance. The sample standard deviation is $S={\sqrt {S^{2}}}={\sqrt {\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{n-1}}}$.

Note that the denominator is $n-1$ instead of $n$.

We can show that $E[S^{2}]=\sigma ^{2}$.

${\begin{array}{rcl}\mathrm {E} [S^{2}]&=&\mathrm {E} \left[{\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{n-1}}\right]\\&=&\mathrm {E} \left[{\frac {\sum _{i=1}^{n}(X_{i}^{2}-2X_{i}{\bar {X}}+{\bar {X}}^{2}}{n-1}}\right]\\&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-2\cdot \sum _{i=1}^{n}E[X_{i}{\bar {X}}]+\sum _{i=1}^{n}E[{\bar {X}}^{2}]\right)\\&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-2\cdot \sum _{i=1}^{n}E\left[X_{i}\left((1/n)\sum _{j=1}^{n}X_{j}\right)\right]+\sum _{i=1}^{n}E\left[\left((1/n)\sum _{j=1}^{n}X_{j}\right)^{2}\right]\right)\\&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-(2/n)\cdot \sum _{i=1}^{n}\sum _{j=1}^{n}E\left[X_{i}\cdot X_{j}\right]+n\cdot E\left[\left((1/n)\sum _{j=1}^{n}X_{j}\right)^{2}\right]\right)\\&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-(2/n)\cdot \sum _{i=1}^{n}\sum _{j=1}^{n}E\left[X_{i}\cdot X_{j}\right]+(1/n)\cdot E\left[\left(\sum _{j=1}^{n}X_{j}\right)^{2}\right]\right)\\\end{array}}$

We note that since $X_{i}$ and $X_{j}$ are independent, we have that

$E[X_{i}X_{j}]=E[X_{i}]E[X_{j}]=\mu \cdot \mu =\mu ^{2}$.

Let's deal with the middle term here.

${\begin{array}{rcl}\sum _{i=1}^{n}\sum _{j=1}^{n}E\left[X_{i}\cdot X_{j}\right]&=&\sum _{i=1}^{n}E[X_{i}X_{i}]+\sum _{i=1}^{n}\sum _{j\neq i}E[X_{i}X_{j}]\\&=&\sum _{i=1}^{n}E[X_{i}^{2}]+\sum _{i=1}^{n}\sum _{j\neq i}E[X_{i}]E[X_{j}]\\&=&nE[X^{2}]+n(n-1)E[X]E[X]\\&=&nE[X^{2}]+n(n-1)\mu ^{2}\\\end{array}}$

Let's work on the third term which ends up being the same as the middle term.

$E\left[\left(\sum _{j=1}^{n}X_{j}\right)^{2}\right]=E\left[\sum _{j=1}^{n}\sum _{k=1}^{n}X_{j}X_{k}\right]=\sum _{j=1}^{n}\sum _{k=1}^{n}E[X_{j}X_{k}]=nE[X^{2}]+n(n-1)\mu ^{2}.$

Let's put everything together:

${\begin{array}{rcl}\mathrm {E} [S^{2}]&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-(2/n)\cdot \sum _{i=1}^{n}\sum _{j=1}^{n}E\left[X_{i}\cdot X_{j}\right]+(1/n)\cdot E\left[\left(\sum _{j=1}^{n}X_{j}\right)^{2}\right]\right)\\&=&{\frac {1}{n-1}}\left(nE[X^{2}]-(2/n)(nE[X^{2}]+n(n-1)\mu ^{2})+(1/n)(nE[X^{2}]+n(n-1)\mu ^{2})\right)\\&=&{\frac {1}{n-1}}\left(nE[X^{2}]-2E[X^{2}]-2(n-1)\mu ^{2}+E[X^{2}]+(n-1)\mu ^{2}\right)\\&=&{\frac {1}{n-1}}\left((n-1)E[X^{2}]-(n-1)\mu ^{2}\right)\\&=&E[X^{2}]-\mu ^{2}=\sigma ^{2}\\\end{array}}$

Summary

Sample means:

${\bar {X}}={\frac {X_{1}+X_{2}+\cdots +X_{n}}{n}}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}$

Sample variance:

$S^{2}={\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{n-1}}$

Properties of sample means and sample variances

• $E[{\bar {X}}]=\mu$
• $Var[{\bar {X}}]=\sigma ^{2}/n$
• $E[S^{2}]=\sigma ^{2}$

Distribution of sample means

While we know basic properties of sample means ${\bar {X}}$, if we want to perform other statistical calculation (i.e., computing confidence intervals or testing hypotheses), it is very useful to know the exact distribution of ${\bar {X}}$.

For a general population, it will be hard to deal the the distribution of ${\bar {X}}$ exactly. However, if the population is normal, we are in a very good shape.

Recall the definition of ${\bar {X}}$:

${\bar {X}}=(X_{1}+X_{2}+\cdots +X_{n})/n.$

Therefore, ${\bar {X}}$ is a sum of independent normally distributed random variables. A nice property of normal random variables is that the sum of normally distributed random variables remains a normal random variable. Since a normal random variable is uniquely determined by its mean and variance, we have the following observation.

Distribution of sample means of normal populations.

If the population is normally distributed with mean $\mu$ and variance $\sigma ^{2}$, the distribution of ${\bar {X}}$ is normal with mean $\mu$ and variance $\sigma ^{2}/n$.

Examples

Ex1. Suppose that the population has mean $\mu =15$ and variance $\sigma ^{2}=15$. If you select a sample of size 20, what is the probability that the sample mean ${\bar {X}}$ is greater than 17?

Solution:

The sample mean ${\bar {X}}$ is normal with mean $\mu =15$ and variance $\sigma ^{2}/n=15/20=0.75$. Therefore,

${\frac {{\bar {X}}-15}{\sqrt {0.75}}}$

is unit normal.

Note that

$P\{{\bar {X}}>17\}=P\{{\frac {{\bar {X}}-15}{\sqrt {0.75}}}>{\frac {17-15}{\sqrt {0.75}}}\}\approx P\{{\frac {{\bar {X}}-15}{\sqrt {0.75}}}>2.309\}$

We can look at the standard normal table and find out that $P\{0\leq Z\leq 2.31\}=0.48956$, for a unit normal random variable Z. Thus, the probability

$P\{{\bar {X}}>17\}\approx 0.5-0.48956=0.0104,$

which is roughly 1%.

Ex2.

To be added...

Why do we use normal distributions?

Normal random variables appear very often in our treatment of statistics. This is not just a coincidence. See limit theorems.