Line Of Best Fit With Perpendicular Error

Question

The standard statistical formula for the least squares error gives us a line that minimises the sum of the vertical distances of the sample points to the line. Suppose that I wanted to find the equation of a line that minimises the sum of the perpendicular distance of the points to the line, is there a way of analytically solving this problem?

Answer 1

This is well-known. See https://fr.scribd.com/doc/14819165/Regressions-coniques-quadriques-circulaire-spherique . Figure below:

Note : For the choice of sign, compute $\sum_{k=1}^n(ax_k+b-y_k)^2$ in both cases and keep the smaller.

Also, see : http://mathworld.wolfram.com/LeastSquaresFittingPerpendicularOffsets.html

Moreover, it is of interest to look at the related Principal Component Analysis method. https://en.wikipedia.org/wiki/Principal_component_analysis

A numerical example of the principal component regression is given in page 12 of this paper : https://fr.scribd.com/doc/31477970/Regressions-et-trajectoires-3D . This example is in 3D. But is is very easy to see how proceed on the same manner in 2D. which is even simpler.

LATTER ADDITION

Numerical example with the principal component method :

Comparison with the above least mean square offset method :

The results of both methods are exactly the same, as it is analytically expected.

Graphical representation :

Answer 2

Just build a function of higher in-dimensionality than whatever you tried to fit firstly. And instead of fitting to values you fit towards a level-set.

So instead of fitting to $$f(x)=kx+m, f(x_k) = y_k$$ We fit to: $$f(x,y) = ax+by+c, f(x_k,y_k) = 0$$

This will automatically handle all your roubles ( if you add some tea at the start ).