An Algebraic Understanding of Degrees of Freedom

66 Views Asked by At

I am looking at Simple Linear Regression where our model has two parameters, i.e. $$ E[Y|x_{i}] = b_{0} + b_{1}x_{i} $$

I am having trouble understanding where the idea that $$ \frac{SSR}{\sigma^{2}}\sim X_{1}^{2} \text{ and } \frac{SSRes}{\sigma^{2}}\sim X_{n-2}^{2} $$

I am going to start off laying out the knowledge that I already have and hopefully you can pick up where I am going wrong.

Here are some quick facts to know that make things easier.

  1. Let $ X\sim N(\mu,\sigma^{2}). $ Then $ \frac{X-\mu}{\sigma} \sim N(0,1) $
  2. Let $ Z\sim N(0,1).$ Then $ Z^{2} \sim X_{1}^{2} $
  3. Let $ C \sim X_{n_{1}}^{2} \text{ and } K \sim X_{n_{2}}^{2}. $ Then $ (C+K) \sim X_{n_{1}+n_{2}}^{2} $
  4. Degrees of Freedom are defined to be the sample size minus the number of estimated parameters.
  5. We assume $ \epsilon_{i}\sim N(0,\sigma^{2}) $ where $\epsilon_{i}$ are the unseen population errors

The first time degrees of freedom started to make sense as a concept was seeing Cochran's Theorem which says $$ \sum \left( \frac{X_{i}-\mu}{\sigma} \right)^{2} = \sum \left( \frac{X_{i}-\bar{X}}{\sigma} \right)^{2} + \left( \frac{\bar{X}-\mu}{\sigma/\sqrt{n}} \right)^{2} $$

$$ X_{n}^{2} = X_{?}^{2} + X_{1}^{2} $$

Of course, a (n-1) belongs where the question mark is. The algebra says something along the lines of "Deviation from the Population Mean = Deviation from the Sample Mean + Bias of $\overline{X}$ from $\mu.$"

Cochran's Theorem seems to be a specific instance of the Bias-Variance Tradeoff and shows the idea that calculating a sample estimator reduces the Degrees of Freedom.

Next, I see how $ SST = SSR + SSRes .$ So $$ \frac{SST}{\sigma^{2}} = \frac{SSR}{\sigma^{2}} = \frac{SSRes}{\sigma^{2}} $$ $$ \left( \frac{y_{i}-\overline{y}}{\sigma} \right)^{2} = \left( \frac{\hat{y_{i}}-\overline{y}}{\sigma} \right)^{2} + \left( \frac{{y_{i}}-\hat{y}_{i}}{\sigma} \right)^{2} $$

From Cochran's Theorem, we know that $ \frac{SST}{\sigma^{2}} \sim X_{n-1}^{2} $ but I have not been able to determine what degrees of freedom for $ \frac{SSR}{\sigma^{2}} $ or $ \frac{SSRes}{\sigma^{2}} $ are.

My next thought was to try messing with the Sum of Squared Errors. $$ \sum \left( \frac{y_{i}-E[Y|x_{i}]}{\sigma} \right)^{2} = \sum \left( \frac{\epsilon_{i}-E[\epsilon_{i}]}{\sigma^{2}} \right)^{2} = \frac{SSE}{\sigma^{2}} \sim X_{n}^{2} $$

Both in Cochran's Theorem and ANOVA, the technique of adding 0 is used. The value being adding must be a sampled estimator though. So I tried adding $ (\overline{y}-\overline{y}) $ and $ (\hat{y_{i}}-\hat{y_{i}}) $ into the SSE expression but this hasn't seemed to work.

So overall, I have two main questions:

  1. How do I show that $ \frac{SSR}{\sigma^{2}} \sim X_{1}^{2} $ or $ \frac{SSRes}{\sigma^{2}} \sim X_{n-2}^{2} $
  2. Are there any ideas worth exploring by partitioning $ \frac{SSE}{\sigma^{2}} $

EDIT:

Found the answer HERE and HERE

The answer is to decompose the Sum of Squared Errors where $$ \frac{SSE}{\sigma^{2}} = \frac{SSRes}{\sigma^{2}} + \left( frac{\overline{\epsilon}}{\sigma/\sqrt{n}} \right)^{2} + \left( \frac{\hat{b_{1}}-E[\hat{b_{1}}]}{\sigma\sum (x_{i}-\overline{x})^{2}} \right)^{2} $$

$$ \sum \left( \frac{\epsilon_{i}}{\sigma} \right)^{2} = \sum \left( \frac{\hat{\epsilon_{i}}}{\sigma} \right)^{2} + \left( \frac{\overline{\epsilon}}{\sigma/\sqrt{n}} \right)^{2} + \left( \frac{\hat{b_{1}}-E[\hat{b_{1}}]}{\sigma\sum (x_{i}-\overline{x})^{2}} \right)^{2} $$

$$ X_{n}^{2} = X_{?}^{2} + X_{1}^{2} + X_{1}^{2} $$

You first have to calculate $ \overline{\epsilon} $ and use the fact that $ \sum(\epsilon_{i}-\overline{\epsilon})^{2} = SSE - n\overline{\epsilon} $