Why does the regression line of $x$ on $y$ and $y$ on $x$ meet at $\bar{x}$ and $\bar{y}$?

Question

Why does the least squares regression line of $x$ on $y$ and $y$ on $x$ intersect at $\bar{x}$ and $\bar{y}$? Also, why are the form of regression lines as they are? For the general form $y-\bar{y}=b(x-\bar{x})$, what is $b$, and how is this derived?

Answer 1

If we have that $y$ is a dependant variable on $x$, then the least squares regression line is given as

$$ y - \bar{y} = \frac{\operatorname{Cov}[x,y]}{\operatorname{Var}[x]} (x-\bar{x}) $$

We can derive this formula by considering the optimization problem of minimizing the square of the residuals; more formally if we have a set of points $(x_1,y_1),(x_2,y_2), \ldots , (x_n,y_n)$ then the least squares regression line minimizes the function

$$ D(a,b) = \sum^n_{i=1} \epsilon_i^2 = \sum^n_{i=1} \left(y_i - [a + bx_i]\right)$$

so by solving for $\partial D / \partial a = 0$ and $\partial D / \partial b = 0$ simulataneously we get the above form.

Given the above form of the least squares regression, it should then become apparent why if we then take a least squares regression where $x$ is dependant on $y$ why the two lines intersect at $(\bar{x},\bar{y})$ - if it isn't immediate obvious, then recall the definition of $\operatorname{Cov}[x,y]$ and wlog. consider the case where $\bar{x}=\bar{y}=0$.