What is *energy* in the context of linear algebra?

Question

I have heard and read the term energy in the context of LA a few times now. e.g:

The algorithm is based on the geometrical observations that the word embeddings (across all representations such as Glove, word2vec etc.) have a large mean vector and most of their energy (after subtracting the mean vector) is located in a subspace of about 8 dimensions.

I am having a hard time finding explanation on this energy concept however. Searching for subspace energy will give you Star Trek related results for example :D Searching for linear algebra energy will result in things like What is the importance of linear algebra in renewable energy engineering?

So, can you explain to me what energy in LA is and how I can find material on it?

Answer 1

It can reasonably be inferred from your quote that the phrasing have most of one's energy in subspace $V$ is an empiric way to say that a vector "almost lies in" $V$.

There are different ways you can measure that. For instance, in a Euclidean space, the ratio $$\frac{\|\text{orthogonal projection of $v$ onto $V$}\|}{\|v\|}$$ somewhat tells you the "proportion" of $v$ that lies in $V$.

A practical framework where this could occur is signal analysis, where the energy of a signal is the sum of squares of its Fourier components.

Answer 2

My best guess is that "energy" here means the squared magnitude of a vector:

$$E(\mathbf{x}) = \|\mathbf{x}\|^2.$$

This would be a generalization of what "energy" means for an analog signal (such as a sound wave, or an electric signal traveling through a long transmission line), which is the integral of the square of the analog signal:

$$E(f) = \|f\|^2 = \int_{-\infty}^\infty f(x)\ dx.$$

The quoted paragraph is describing a collection $\mathbf{w}$ of vectors and stating that "most of their energy (after subtracting the mean vector) is located in a subspace of about 8 dimensions," meaning that $\mathbf{w}$ can be written as $\mathbf{x} + \mathbf{y} + \mathbf{z},$ where

• $\mathbf{x}$ is "the mean vector,"
• $\mathbf{y}$ is the component of $\mathbf{w}$ "after subtracting the mean vector" which is found in a subspace of about 8 dimensions,
• $\mathbf{z}$ is the remainder,
• $\mathbf{x}$, $\mathbf{y}$, and $\mathbf{z}$ are pairwise orthogonal,
• as a result of the above point, $E(\mathbf{w}) = E(\mathbf{x}) + E(\mathbf{y}) + E(\mathbf{z}),$ and
• $E(\mathbf{y})$ is much greater than $E(\mathbf{z})$.