Say I have $n$ i.i.d. samples $x_1,\cdots, x_n$ from a normal distribution $N(\mu, \sigma^2)$. I want to estimate the mean $\mu$ using the hard thresholding estimator $\mu^* = \bar{x}1_{|\bar{x}| \ge \alpha}$, i.e. $\mu^* = \bar{x}$ if $\bar{x}$ is $\ge \alpha$ or $\le -\alpha$ and $\mu^* = 0$ otherwise. I am trying to find the optimal $\alpha$ such that $E(x - \mu^*)^2$ is minimized.
So $$ \begin{aligned} E(x - \mu^*)^2 &= E(x - \mu + \mu - \mu^*)^2 \\ &= E[(x-\mu)^2] + E[(\mu - \mu^*)^2] \\ &= \sigma^2+ \mu^2 + \mu^2 + \sigma^2/n - \int_{-\alpha}^\alpha x^2p(x)dx - 2\mu^2 + 2\mu\int_{-\alpha}^\alpha xp(x)dx \\ & = \sigma^2+\sigma^2/n - \int_{-\alpha}^\alpha x^2p(x)dx + 2\mu\int_{-\alpha}^\alpha xp(x)dx \\ &=\sigma^2+\sigma^2/n + \int_{-\alpha}^\alpha (2\mu x- x^2)p(x) dx \end{aligned} $$ where $p(x)$ is the normal density with mean $\mu$ and variance $\sigma^2/n$ (the density of $\bar{x}$). If I differentiate this with respect to $\alpha$ and set that to $0$, I get $$ \begin{aligned} (2\mu\alpha-\alpha^2)p(\alpha) + [2\mu(-\alpha)-\alpha^2]p(\alpha) = -\alpha^4p(\alpha) = 0 \end{aligned} $$ But since normal density is an exponential function, which is non-zero, does this imply that only $\alpha = 0$ satisfies the above equation? This does not seem right to me. Can someone point out what I did wrong? Thanks in advance!
EDIT: Thanks to the comment by @Andreas Lenz, I realized that I mistakenly use $p(α)=p(−α)$ in the last step, so my last step should really be $(2\mu \alpha − \alpha^2)p(\alpha)-[2\mu\alpha+\alpha^2]p(-\alpha) = 0$. $\alpha = 0$ and $\alpha = \infty$ are certainly two solutions to the equation, but I am just wondering whether there exists a closed form of the minimizer other than these two extreme values, since setting $\alpha = 0$ or $\alpha = \infty$ defeats the purpose of my estimator. Any suggestions or hints would be appreciated. Thanks!
Your mistake is within the differentiation. If you differentiate the MSE with respect to $\alpha$, you obtain $$ (2\mu\alpha-\alpha^2)p(\alpha) + (2\mu(-\alpha)-(-\alpha)^2)p(-\alpha) = (2\mu\alpha-\alpha^2)p(\alpha) - (2\mu\alpha+\alpha^2)p(-\alpha). $$ Notice that you cannot use $p(\alpha)=p(-\alpha)$ here, since $\mu$ is not necessarily $0$.
You further need to take care when searching for the minimizer $\alpha^*$ as it might also be equal to $0$ or $\infty$. Thus the derivative does not necessarily vanish at the minimizing point.