I encountered an interesting paper with the following loss function:
Now they say in the paper that the minimum with respect to the right hand sight with respect to $\sigma^2$ can be obtained by setting $\sigma^2$ to $|\hat{y}_i-y_i|$. My math is a bit rusty, can someone explain why this is the case?
You can just take the derivative with respect to $\sigma_i^2$ on the RHS and equate this to 0. The derivative is equal to: \begin{align*} \frac{\partial \ell}{\partial \sigma_i^2} &= \frac{1}{2}\left(\frac{1}{\sigma_i^2} - \frac{(\hat{y}_i - y)^2}{\sigma_i^4}\right). \end{align*} Now equating this to 0 yields \begin{align*} \frac{1}{\sigma_i^2} = \frac{(\hat{y}_i - y)^2}{\sigma_i^4} \end{align*} which implies $\sigma_i^2 = (\hat{y}_i - y)^2$ and thus $\sigma_i = |\hat{y}_i - y|$.