fundamental error and variance decompositions: bias + variance and explained/non-explained variance

46 Views Asked by At

As I understand there are two fundamental decompositions behind linear regression

  1. the general bias + variance decomposition

  2. total variance = explained variance + non-explained variance

how do these two relate to each other?

\begin{equation} \text{MSE} = \operatorname{E}_x\bigg\{\operatorname{Bias}_D[\hat{f}(x;D)]^2+\operatorname{Var}_D\big[\hat{f}(x;D)\big]\bigg\} + \sigma^2. \end{equation}

can we say that $\operatorname{Var}_D\big[\hat{f}(x;D)\big]$ is the explained variance and $\sigma^2$ is the non-explained variance?

when the model "is correct", i.e., data is produced by the same model considered in the linear regression, is it the case that the following holds?

\begin{equation} \text{MSE} = \operatorname{E}_x\bigg\{ 0 + \sigma^2 \bigg\} + \sigma^2 = 2 \sigma^2 \end{equation}

References

https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

http://rasbt.github.io/mlxtend/user_guide/evaluate/bias_variance_decomp/