We have learned that the closed form solution:
$$(X^T*X)^{-1}*X^T*\vec{y} = \vec{w}$$
for linear regression with X the $n*d$ design matrix, y the $n*1$ output and w the $d*1$ weight vector is only attainable, if $X^T*X$ is invertible. Invertibility means that $X^T*X$ has full rank, which in turn means that X has full rank. However only quadratic matrices can have full rank - does that mean that the closed form solution is only given for quadratic matrices?
No, not only square matrices can have full rank. A $n \times d$ matrix $X$ is said to have full rank if $rank(X) = \min \{n,d\}$
Consider $X = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$, this matrix has full rank, and $X^t X = [2]$ is invertible.