This question is similar with my previous question, but it seems more difficult.
I'm given a function $0.5(y-\theta)^2 + 0.5 \lambda I(|\theta| \neq 0) $, $\lambda \geq 0$ that should be minimized with respect to $\theta$.
The result should be $\hat \theta = y I(|y| >\lambda)$.
I don't understand why we have an indicator function. I would expect that we consider two cases (when $\theta \neq 0$ and $\theta = 0$) and just take derivatives and assign them to zero. Why is it wrong and how can I obtain the result above?
Let \begin{align}f(\theta; y, \lambda ) &= 0.5(y-\theta)^2 + 0.5 \lambda I (|\theta| \ne 0)\\ &= \begin{cases} 0.5y^2 &, \theta = 0 \\0.5(y-\theta)^2 + 0.5\lambda &, \theta \ne 0\end{cases} \end{align}
If $y=0$, then we select $\theta=0$ and obtain an objective function of $0$. Notice that if $\theta \ne 0$, then $f$ is positive. Hence the optimal $\theta$ is $0$.
Now we consider the case when $y \ne 0$. The minimum when we only consider $\theta \ne 0$ would be $0.5 \lambda$ when we let $\theta$ be $y$. When $\theta=0$, the minimum value is $0.5y^2$. Hence we have to consider which is smaller.
We study when does
$$f(y) < f(0)$$ $$\iff 0.5\lambda < 0.5y^2$$ $$\iff\lambda < y^2$$ $$\iff \sqrt{\lambda}< |y|$$
Hence \begin{align}\hat{\theta}&=\begin{cases} y &, \sqrt{\lambda}<|y| \\ 0&, \sqrt{\lambda} \ge |y|\end{cases}\\&=yI(|y| >\sqrt{\lambda}) \end{align}
If $\sqrt{\lambda}=y$, we can actually let $\hat{\theta}$ takes value $y$ as well.