Dimension of matrix in polynomial regression

Question

I'm having trouble understanding that when you minimize a polynomial regression

$$ \min_\beta \left\{ \left \| \frac{1}{2} (y - B\beta )\right \| ^2 \right\} $$

$ B\beta = y $, the B is a (m + 1) x (m + 1) dimension matrix? Where m is the highest exponent in the equation (deg( polynomial ) = m)

Like does the fact that when $ \beta $ is minimized you would need $ \geq$ (m+1) equations to solve for $ \beta $?

If there is a resource that someone could please point me to or a proof showing that this is true, I'd really appreciate it, thank you!

Answer 1

If $B$ is a squared matrix of $(m+1)\times (m+1)$ then you'll have a unique solution (assuming that you don't have any redundant observations). No OLS is needed. You can still minimize and you'll get the same answer, bu the minimization is meaningless in this case. The more common situation is when $B$ is of order $n\times p$, where $n > p$. In this case, the system of linear equations $$ B\beta = y, \quad (1) $$ is "overdetermined", namely you have much more equations $(n)$ than unknown parameters ($p$). In this case, you have to find an approximate solution which can be found by solving the following modified linear system (it is the first order condition of your minimization problem), $$ B'B\beta = B'y, \quad (2) $$ here you'll have $p$ equations with a unique solution $\hat{\beta} = (B'B)^{-1}B'y$. So, by solving $(2)$ you get the "best" solution of $(1)$ which was your inital goal.

Answer 2

To understand, you must first consider the polynomial $$y = b_0 + b_1x+ b_2x^2 + \cdots + b_mx^m$$ The equation can be written as $y=B\beta$ where $B$ is the matrix of coefficients $$B = \begin{bmatrix}b_0&0&\cdots&0\\0&b_1&\cdots&0\\\vdots&\vdots&\ddots&\vdots\\0&\cdots&0&b_m\end{bmatrix}$$ and $\beta = [1\;x\;x^2\;\cdots\;x^m]$, a vector of the exponents.

Note there are an $m+1$ number of coefficients, thus the coefficient matrix ends up being $(m+1)\times(m+1)$.