I was just wondering what happens generally if i send all my x points to y's and y's to x's (i.e reflect along the y=x line) - if I change the x's and y's will my old error minimizing line still be the error minimizing line after reflection? How about if I rotate points?
Many thanks
You address here a very important point. If you consider the regression of $y=a+bx$, you minimize $$F(a,b)=\sum_{i=1}^n(a+bx_i-y_i)^2$$ while if you consider the regression of $x=c+dy$, you minimize $$G(c,d)=\sum_{i=1}^n(c+dy_i-x_i)^2$$ If you look at the expressions for the coefficients, you will notice that $b\times d\neq 1$.
For illustration purposes, let me take four points $(x_i=i,y_i=3+5i+(-1)^i,i=1,\cdots,4$) and perform the regressions. As a result, we get $y=2.000+5.400\, x$ and $x=-0.493+0.196 \,y$ (inversing the last would give $y=2.512+5.096\, x$) and, as said, the product of the slopes is $1.05965$. So, plotted on the same graph, the lines will not coincide at all.
The main reason for that is that, when performing classical least square regression, we assume no error for the independent variable and error for the dependent variable.
I hope this helps but, if this does not clarify, please post.