Confused about the terminology of least regression squares

Question

I am confused by the language of my math text book where it says $\hat y = \alpha \times$basis function summation from $i=1$ to $n$? What are basis functions? And why does it says there are more data points than basis functions? I know how to derive $\alpha$ using linear algebra and calculus by using $mx + c$ as the equation of the regression line.

Also I am a bit confused on the fact that how does it know that the smaller distance of the actual data point and $\hat y$ is perpendicular to the line of $\hat y$? I will be very very grateful if anyone can help me with this.

Answer 1

A model which is linear with respect to the parameters write $$y=\sum_{k=1}^n a_k\,f_k(x)$$ and the $f_k(x)$ are the $k$ basis functions.

Suppose that you want to fit data to the model $$y=a_0+a_1\sin(x)+a_2\log(x)+a_3 e^{-\pi x}$$ Define $t_i=\sin(x_i)$, $u=\log(x_i)$, $v=e^{-\pi x_i}$. So the model is just $$y=a_0+a_1t+a_2u+a_3v$$ which corresponds to a multilinear regression.

Answer 2

You are approximating the data points with a linear combination of independent functions. In the case of the straight line of equation

$$y=mx+p$$

there are two functions which are the identity,

$$y=x$$ and the unit,

$$y=1.$$

You can generalize with more functions, such as the powers $x^2,x^3,\cdots$ or any other. Basis function comes from the terminology of linear algebra.

Every data point lets you write an equation, such as

$$(x_k,y_k)\to y_k=mx_k+p.$$

When there are more points than basis functions, i.e. than unknown coefficients, the system is overdeterminate and has no solution. In this case, you try to find a good compromise, i.e. one that approximately satisfies all equations.

---

Notice that the minimization via the model $y=mx+p$ does not achieve the minimum distance to the lines, but the vertical distance to the line. See total least squares for the former. https://en.wikipedia.org/wiki/Total_least_squares