Hi I was going through the MIT 2005 Machine Learning homework assignments and I was having trouble understanding a few concepts in probability theory. I would be obliged if anyone could validate my answers, or if I am wrong, point me in the right direction.
Here is a link to the original problem set: http://www.slideshare.net/adimanunited/ps1-v3-r
We have a set of unidimensional data, $X1, . . . , Xn$, drawn from the positive reals. We will consider two different models for its distribution:
• Model 1: The data are drawn from a uniform distribution on the interval $[0, b]$. This model has a single positive real parameter, $0 < b$.
• Model 2: The data are drawn from a uniform distribution on the interval $[a, b]$. This model has two positive real parameters, $a$ and $b$, such that $0 < a < b$.
Question 0: What’s the mean of each of the distributions?
My answer: $b/2$ for the first one, $(a+b)/2$ for the second one.
In model 1, we just need to consider the distribution of $f_{\hat b}$. Generally speaking, the pdf of the maximum of a set of data drawn from pdf $f$, with cdf $F$, is $f_{\hat b} (x) = nF(x)^{n−1}f(x) $.
The idea is that, if $x$ is the maximum, then $n − 1$ of the other data values will have to be less than $x$, and the probability of that is $F(x)^{n−1}$, and then one value will have to equal $x$, the probability of which is $f(x)$. We multiply by $n$ because there are $n$ different ways to choose the data value that could be the maximum.
Question 1: What is $f_{\hat b}$ in the particular case where the data are drawn uniformly from $0$ to $b$?
My answer: $f_{\hat b}=1/b$
Question 2: Write an expression for the expected value of $\hat \mu$, as an integral.
My answer: $$ \int ^{\infty}_{-\infty} x{f_{\hat b}(x)} ~ \mathrm{d}{x}. $$
Question 3: What is the squared bias of $\hat \mu$? Is this estimator unbiased? Is it asymptotically unbiased?
I know $bias^2(\hat \theta) = (ED[\hat \theta] − \theta)^2$ but not able to move ahead.
Question 4:Write an expression for the variance of $\hat \mu$, as an integral.
I know I can get the answer as $$ \int ^{\infty}_{-\infty} (x-\mu)^2{f_{\hat b}(x)} ~ \mathrm{d}{x}. $$ but the actual result should be $V [\hat \mu]=b^2 n/(4 (n+1)^2 (n+2))$ and I get a totally different value.
Question 5: What is the mean squared error of $\hat \mu$?
I know that $MSE=bias^2+var$ but I am not able to get any of those.
Kindly help.
This is the answer for the first part only. Our estimate of the parameter $b$ is $$\hat b = \max_i X_i,$$ the largest observation in our sample. We are told that the density of $\hat b$ is given by $$f_{\hat b}(x) = nF_X(x)^{n-1} f_X(x),$$ where $F_X$ and $f_X$ are the CDF and density of each of the $X_i$s; i.e., it is the CDF and PDF of a (continuous) uniform distribution. If this distribution is uniform on $[0,b]$, then clearly $$f_X(x) = \frac{1}{b}, \quad F_X(x) = \frac{x}{b}, \quad 0 < x < b.$$ Therefore, the PDF of $\hat b$ is $$f_{\hat b}(x) = n \biggl( \frac{x}{b} \biggr)^{n-1} \frac{1}{b} = \frac{nx^{n-1}}{b^n}.$$
Try to think carefully about how the other questions can be answered in light of the above.