Geometry of underdetermined regression loss function in $n$-dimensions

Question

Let's say I have a regression loss function defined as $(AX-y)^2$, where $A \in \mathbb{R}^{m \times n}$, $X \in \mathbb{R}^{n \times 1}$ and $y \in \mathbb{R}^{m \times 1}$

If $A$ is well determined, that is $m \geq n$, then in the case where $n=2$, the shape of the loss function is a concave bowl, with an unique minima at the bottom of the bowl, as illustrated in an example Figure below:

.

However, if $A$ is under-determined, that is $m \lt n$, there will be a set of minima instead of an unique one. In the case where $n=2$ and $m=1$, we get a trough shape where the set of minima lies on a line, as illustrated in the example Figure below:

Now my problem is as such:

• I know for well determined case, there is always an unique minima solution. The level set (contour/surface etc. where the value of the loss function is the same), when $n=2$ is an ellipse, $n=3$ is an ellipsoid, and $n=d$ is a $d$-dimensional hyper-ellipsoid.
• I know in an under-determined case, no unique minima solution exist. The set of solutions for $n=2$ when $m=1$ lies on a line, and if we fix at one solution, the loss function as we move away from that point is a parabola, and the level set is just 2 points lying on the parabola at the same height.
• Does this imply that
  • at higher $n$, the solution set lies on the line/plane/hyper-plane determined by the degree of freedom $n-m$. For e.g. when $n=3$ and $m=2$, the minima solution lies on some line, but when $m=1$, then the solution lies on some plane.
  • at higher $n$ when $m \lt n$, if we choose to fix at one minima solution, the loss function centered around that position is a $m$-$d$ parabola, and the level set is a $(m-1)$-$d$ ellipse. For e.g. when $n=3$ and $m=2$, the loss function centered around any particular solution on the solution line is a paraboloid and the level set is an ellipse (just like the first Figure). The shape of the full loss function is like an elliptic cylinder, where if you slice it anywhere, you get a ellipse that is the contour map of the loss centered at one particular minima.