ผลต่างระหว่างรุ่นของ "Probstat/notes/parameter estimation"

จาก Theory Wiki
ไปยังการนำทาง ไปยังการค้นหา
 
(ไม่แสดง 11 รุ่นระหว่างกลางโดยผู้ใช้คนเดียวกัน)
แถว 61: แถว 61:
 
<math>\hat{\theta} = \arg\max_\theta f(x_1,x_2,x_3,\ldots,x_n|\theta)</math>.
 
<math>\hat{\theta} = \arg\max_\theta f(x_1,x_2,x_3,\ldots,x_n|\theta)</math>.
 
</center>
 
</center>
 +
 +
=== Examples ===
  
 
'''EX1''': A box contains 5 balls; they are blue balls and white balls.  We randomly choose a ball from the box 3 times with replacement and get 2 blue balls and 1 white balls.  Estimate the number of blue balls in the box using the maximum likelihood method.
 
'''EX1''': A box contains 5 balls; they are blue balls and white balls.  We randomly choose a ball from the box 3 times with replacement and get 2 blue balls and 1 white balls.  Estimate the number of blue balls in the box using the maximum likelihood method.
แถว 82: แถว 84:
 
! 5
 
! 5
 
|-
 
|-
| <math>P(outcome|\theta)</math>
+
| Probability
 
| 0
 
| 0
 
| 0.096
 
| 0.096
แถว 91: แถว 93:
 
|}
 
|}
  
 +
Since the probability is maximum at <math>\theta=3</math>, the maximum likelihood estimator for the number of blue balls is 3.
 +
 +
'''EX2''': Maximum likelihood estimator for a Bernoulli parameter
 +
 +
A coin has probability <math>p</math> of turning up heads.  We perform <math>n</math> independent trials and obtain random variables <math>X_1,X_2,\ldots,X_n</math> such that <math>X_i=1</math> if in the ''i''-th trial, we get a Head, and <math>X_i=0</math> otherwise.  What is the maximum likelihood estimator for <math>p</math>?
 +
 +
'''Solution:'''  Note that <math>P\{X_i=1\} = p = 1 - P\{X_i=0\}</math>.  We rewrite the probability that the random variable <math>X_i</math> takes value <math>x_i</math> for <math>x_i=0,1</math> as
 +
 +
<center>
 +
<math>P\{X_i = x_i\} = p^{x_i}(1-p)^{1-x_i}</math>.
 +
</center>
 +
 +
('''Remarks:''' The random variable is <math>X_i</math>.  We denote the actual value of the variable with <math>x_i</math> (i.e., with a small x).)
 +
 +
Therefore, the probability of obtaining the data <math>x_1,x_2,\ldots,x_n</math> given parameter <math>p</math> is
 +
 +
<center>
 +
<math>f(x_1,x_2,\ldots,x_n|p)
 +
= p^{x_1}(1-p)^{1-x_1}\cdot p^{x_2}(1-p)^{1-x_2} \cdots p^{x_n}(1-p)^{1-x_n}
 +
= p^{\left(\sum_i x_i\right)}(1-p)^{\left(\sum_i (1-x_i)\right)}.
 +
</math>
 +
</center>
 +
 +
Since <math>f(x_1,x_2,\ldots,x_n|p)</math> maximizes also when <math>\ln f(x_1,x_2,\ldots,x_n|p)</math> is maximum, we can take a logarithm of the above term and get
 +
 +
<center>
 +
<math>\ln f(x_1,x_2,\ldots,x_n|p)
 +
= \left(\sum_i x_i\right)\ln p + \left(\sum_i (1-x_i)\ln(1-p)\right)
 +
= \left(\sum_i x_i\right)\ln p + \left(n - \sum_i x_i\right)\ln(1-p).
 +
</math>
 +
</center>
 +
 +
To find its maximum, we differentiate the term and get
 +
 +
<center>
 +
<math>\frac{d}{dp}\ln f(x_1,x_2,\ldots,x_n|p)
 +
= \frac{\sum_i x_i}{p} - \frac{n - \sum_i x_i}{1-p}.
 +
</math>
 +
</center>
 +
 +
Setting <math>\frac{d}{d\hat{p}}\ln f(x_1,x_2,\ldots,x_n|\hat{p}) = 0</math>, we solve the equation and get that
 +
 +
<center>
 +
<math>\frac{\sum_i x_i}{\hat{p}} = \frac{n - \sum_i x_i}{1-\hat{p}},
 +
</math>
 +
</center>
 +
 +
implying that
 +
 +
<center>
 +
<math>\hat{p} = \frac{\sum_i x_i}{n},
 +
</math>
 +
</center>
  
'''EX2''': maximum likelihood estimator for a Bernoulli parameter
+
which is the maximum likelihood estimator for <math>p</math>.
  
 
'''EX3''': maximum likelihood estimator for the Poisson mean
 
'''EX3''': maximum likelihood estimator for the Poisson mean
 +
 +
: ''To be added...''
  
 
== Bayes estimators ==
 
== Bayes estimators ==
 +
Let the observed data be <math>x_1,x_2,\ldots,x_n</math>.  We fist compute the conditional probability
 +
 +
<center>
 +
<math>
 +
f(p|x_1,x_2,\ldots,x_n)
 +
= \frac{f(x_1,x_2,\ldots,x_n,p)}{f(x_1,x_2,\ldots,x_n)}
 +
= \frac{f(x_1,x_2,\ldots,x_n|p)f(p)}{f(x_1,x_2,\ldots,x_n)}.
 +
</math>
 +
</center>
 +
 +
With that distribution function, the Bayes estimator is
 +
 +
<center>
 +
<math>
 +
E[p|X_1=x_1,X_2=x_2,\ldots,X_n=x_n].
 +
</math>
 +
</center>

รุ่นแก้ไขปัจจุบันเมื่อ 18:23, 6 ธันวาคม 2557

This is part of probstat.

Previously we tried to estimate the population means and variances using the sample means and variances. In this section, we shall see the justification why what we did makes sense.

There are many ways to estimate parameters.

Method of moments estimators

See also wikipedia article.

This is probably the simplest estimators. However, they are often biased (as we shall show in the example).

Definition: For a random variable , is called the k-th moment of . Note that the first moment is the mean . The variance of a random variable depends on the first and the second moments.

If we want to estimate a parameter , using the method of moments, we start by writing the parameter as a function of the moments, i.e.,

We then estimate the sample moments

for . Our estimate is thus

EX1: We show how to estimate the variance with the method of moments. Recall that the variance

We first estimate the first moment and the second moment . The estimator is

Note that the estimate is biased, because .

As this example shows, other estimation techniques are usually preferred over the method of moments.

Maximum likelihood estimators

Suppose that we want to estimate parameter based on observations (or sample) . If we can find a joint-distribution function

,

that gives the probability that we observe given a particular value . The maximum likelihood estimator is such that is maximum, i.e.,

.

Examples

EX1: A box contains 5 balls; they are blue balls and white balls. We randomly choose a ball from the box 3 times with replacement and get 2 blue balls and 1 white balls. Estimate the number of blue balls in the box using the maximum likelihood method.

Solution: Let denote the number of blue balls. The possible values are 0,1,2,3,4, and 5. For each value of the probability that we observe the outcome of getting 2 blue balls and 1 while balls is

.

We put the probabilities in the following table.

0 1 2 3 4 5
Probability 0 0.096 0.288 0.432 0.384 0

Since the probability is maximum at , the maximum likelihood estimator for the number of blue balls is 3.

EX2: Maximum likelihood estimator for a Bernoulli parameter

A coin has probability of turning up heads. We perform independent trials and obtain random variables such that if in the i-th trial, we get a Head, and otherwise. What is the maximum likelihood estimator for ?

Solution: Note that . We rewrite the probability that the random variable takes value for as

.

(Remarks: The random variable is . We denote the actual value of the variable with (i.e., with a small x).)

Therefore, the probability of obtaining the data given parameter is

Since maximizes also when is maximum, we can take a logarithm of the above term and get

To find its maximum, we differentiate the term and get

Setting , we solve the equation and get that

implying that

which is the maximum likelihood estimator for .

EX3: maximum likelihood estimator for the Poisson mean

To be added...

Bayes estimators

Let the observed data be . We fist compute the conditional probability

With that distribution function, the Bayes estimator is