I have the following problem:
Let $X_n$ be the maximum of a random sample $Y_1,...,Y_n$ from the density $f(x)=2(1-x), x\in [0,1]$. Find constants $a_n,b_n$ so that $b_n(X_n-a_n)$ converges in distribution to a non-degenerate limit.
I did the following:
Calculate $F(x)=x(2-x), x\in [0,1]$ and find distribution of $X_n$: $$ F_{X_n}(x) = \mathbb{P}[max{Y_i} \leq x] = \mathbb{P}[Y_1 \leq x,...,Y_n \leq x] = \mathbb{P}[Y_1 \leq x]^n = x^n(2-x)^n$$
So
$$ \mathbb{P}[b_n(X_n-a_n) \leq x] = \mathbb{P}[X_n \leq \dfrac{x}{b_n}+a_n] = F_{X_n}(\dfrac{x}{b_n}+a_n)=(\dfrac{x}{b_n}+a_n)^n (2-\dfrac{x}{b_n}-a_n)^n$$
My intuition here is to use that $(1+\dfrac{x}{n})^n \rightarrow e^x$ and choose $a_n=1 , b_n=n$
In that case $$ \mathbb{P}[b_n(X_n-a_n) \leq x] \rightarrow e^xe^{-x}=1$$ However if I understand correctly a "non-degenerate" limit means that it should not be a constant.
Can someone please correct me? What should $a_n,b_n$ be?
As what you have calculated, $$F_{X_n}(x) = [1 - (1-x)^2]^n$$
which will converge to $0$ for $0 \leq x < 1$. So intuitively $X_n$ converge to $1$ in distribution. In order for the limit of $b_n(X_n - a_n)$ to be non-degenerate, $a_n = 1$.
Consider the distribution of $b_n(X_n - 1)$ with $b_n > 0$. As the support of $X_n$ is $[0, 1]$, the support of $X_n - 1$ is $[-1, 0]$ and thus the support of $b_n(X_n - 1)$ is $[-b_n, 0]$. Its CDF is
$$ \begin{align} F_{b_n(X_n-1)}(x) &= \Pr\{b_n(X_n-1) \leq x\} \\ &= \Pr\left\{X_n \leq \frac {x} {b_n} + 1\right\} \\ &= \left[1 - \left(1- \frac {x} {b_n} - 1\right)^2\right]^n \\ &= \left(1 - \frac {x^2} {b_n^2}\right)^n \end{align}$$
So here we require $b_n^2$ to go to infinity in the same order of $n$, in order the function to converge to the exponential function $e^{-x^2}$. We can take $b_n = \sqrt{n}$
The result is $$ \lim_{n\to\infty} F_{\sqrt{n}(X_n - 1)}(x) = \lim_{n\to\infty}\left(1 - \frac {x^2} {n}\right)^n = e^{-x^2}, x < 0 $$
i.e. $\sqrt{n}(X_n - 1)$ converge in distribution, with the CDF
$$ F(x) = \begin{cases} e^{-x^2} & \text{if} & x < 0 \\ 1 & \text{if} & x \geq 0\end{cases}$$
So this is the negative of a Weibull distribution with $k = 2$ and $\lambda = 1$, which is one of the three extreme value distribution. If you take $b_n = -\sqrt{n}$ you get the regular Weibull distribution.