what's the geometric interpretation of the quadratic form by itself?

Question

original question

I understand that the equation $\vec{x}^\top\mathbf{A}\vec{x} = r$ represents an ellipsoid, i.e. the solution space of that equation is an ellipsoid, and you can do the eigen-decomposition of $\mathbf{A}$ to tell you about the principle axis of the ellipsoid. My question is how can you interpret the left-hand side of that equation alone, i.e. the quadratic form by itself?

This comes up during PCA, where we are trying to maximize the quadratic form $\vec{x}^\top\mathbf{A}\vec{x}$ under the constraint that $\vec{x}^\top\vec{x} = 1$. My gut feeling is that the quadratic form under constraint will form the "same" (or probably "similar" is a more appropriate word) ellipsoid as the equation form $\vec{x}^\top\mathbf{A}\vec{x} = 1$, and the $\vec{x}$ to maximize the quadratic form would just be the long axis, but I couldn't prove this to myself nor come up with a formal way to set up this picture in my mind. Sorry in advance if this has been asked before (but I couldn't find it)... Any help would be much appreciated! Thank you!

comments

The original question is sort of vague, and the question "why do the eigenvalues/vectors maximize the quadratic term?" is indeed asked elsewhere, probably several times. Apologize again for the duplication. However, I do realize I was picturing something else that's not found in other questions. I will try to formalize that in the following new question:

new question

Let $\mathbf{A}$ be a positive definite matrix, so that $\vec{x}^\top\mathbf{A}\vec{x} = 1$ represent an ellipsoid. Let $\vec{v}$ be a vector that's on the same direction as $\vec{x}$, and at the same time assume the length of the quadratic form: $\lVert\vec{v}\rVert = \vec{x}^\top\mathbf{A}\vec{x}$. My question is what would the graph of $\vec{v}$ look like? Can you put the unit circle $\vec{x}^\top\vec{x} = 1$, the ellipsoid $\vec{x}^\top\mathbf{A}\vec{x} = 1$, and $\vec{v}$ in the same plot and develope some intuitive geometric relationship between them? (just to be clear, the vector $\vec{v}$ is what I was picturing when I say "geometric interpretation of the quadratic form by itself" in the title)

Answer 1

A quadratic norm is a function $f(x) = \mathbf{x^\top A x}$, which will be a quadratic function of $x_1, x_2, ... x_n$. This graph will look differently depending on how those "parabolas fit together" (this will depend on the matrix $\mathbf{A}$). If $\mathbf{A}$ is positive definite you can view this as a generalisation of a 2D parabola pointing upwards (the function $f$ will be positive for any value $x_1, x_2, ... x_n$, i.e. $\mathbf{x^\top A x} > 0$), so you get an elliptic paraboloid. Similarly with negative definite $\mathbf{A}$. Otherwise you will get a hyperbolic paraboloid (saddle). The cross section of elliptic paraboloid is an ellipse and a cross section of hyperbolic paraboloid is a hyperbola.

Answer 2

First, some background:

Call a set abc (nonce use) if it is absorbing, balanced, and (absolutely) convex. A norm is the "same thing" as an abc set: for any norm, the unit ball $\{x|\lVert x\rVert <1\}$ is abc, and for any abc set, the Minkowski functional is a norm.

---

Now suppose that $A$ and $B$ are abc sets, and let $\lVert\quad\rVert_A$, $\lVert\quad\rVert_B$ be the corresponding norms. Then for any nonzero vector $\mathbf{x}$, $\frac{\lVert \mathbf{x}\rVert_A}{\lVert \mathbf{x}\rVert_B}$ is the proportion $\lambda$ we need to dilate $A$ about the origin so that $B$ and the dilated $\lambda A$ coincide on the line $L_{\mathbf{x}}$ passing through $\mathbf{0}$ and $\mathbf{x}$:

$$\lambda A\cap L_{\mathbf{x}}= B\cap L_{\mathbf{x}}\text{.}$$

It's probably useful to draw the case where $A$ is a disk and $B$ is a filled ellipse.

Answer 3

It may not be a well phrased question but it's a good one! I think the best intuition comes from thinking of a 2d vector space, and then imagine the quadratic form as the height of the landscape as you move around, like a topographic map.

The condition $x^T A x=1$ is a constant-height contour,and for positive definite A that will be an ellipse. The condition $x^Tx=1$ is of course just the unit circle, and so $x^T A x$ is maps a point on the unit circle to the height of the landscape at that point.

To understand why the eigenvectors give the major and minor axes of the ellipse you consider trying to find a minimal or maximal height as you move along the unit circle. As you move around the circle you'll know you're at a max or min when the slope at that point is orthogonal to the your path on the circle. If it's not orthogonal then it means you will either gain or lose height by continuing to move on your path. So as you move around $x$ you're looking for a point where the slope is orthogonal to the unit circle, i.e. parallel to $x$. Now the slope is given by the gradient of the quadratic form which is $A x$ (up to a factor of 2). Therefore what we're looking for is a place where $A x \propto x$, i.e. an eigenvector.