Two ways of interpreting matrix-vector multiplication?

Question

I have been trying to get an intuitive grasp of the matrix-vector multiplication operation.

So far, I've consumed both 3Blue1Brown's videos on this topic as well as studied Gilbert Strang's textbook chapters relevant to matrix-vector multiplication. Both seem to offer two very different ways of intuitively looking at the product.

3Blue1Brown interprets the matrix as transforming the vector by performing operations such as "stretching", "shrinking", "rotating", etc. In other words, the thing being transformed is the vector.

Strang, on the other hand, interprets this operation as the columns of the matrix being added together with the elements of the vector acting as coefficients. In other words, the things being transformed are the columns of the matrix.

I see these as two very different interpretations of the same thing. Is there some way to reconcile these two things? Am I missing something painfully obvious?

Answer 1

Let's talk about $(\color{red}{M}\color{blue}{v})_i=\sum_j\color{red}{M}_{ij}\color{blue}{v}_j$. 3Blue1Brown thinks of this as something the red does to the blue; Gilbert Strang things of it as something the blue does to the red. This pluralism is no more mysterious than wondering whether $\color{red}{3}\times\color{blue}{2}$ is tripling 2 or doubling 3.

Answer 2

Notations:

• Let $M$ be a $m\times n$ matrix consisting of column vectors $\textbf{m}_1, ..., \textbf{m}_n$.
• Let $\textbf v=[a_1, ..., a_n]^{\textsf T}$.
• Let $I$ be the $n\times n$ identity matrix, with columns $\textbf e_i = [0, ...,0, \underbrace{1}_{i-\text{th}}, 0, ..., 0]^{\textsf T}$.

Combining the vectors $\textbf m_i$ using scalars $a_i$ yields the vector $M\textbf v = \sum_i a_i\textbf m_i$. How do we view this sum as the image of $\textbf v$ under transformation? Consider the following facts:

1. $M$ maps $\textbf e_i$ to $\textbf m_i$. (Indeed, we can check the equation $MI=M$ along columns.)
2. $\textbf v = \sum_i a_i\textbf e_i$
3. $M\textbf v =\sum_i a_iMe_i=\sum_i a_i\textbf m_i$.

What "2" and "3" say is: the way $\{\textbf e_i\}$ combine to get $\textbf v$, is the same way as $\{\textbf m_i\}$ combine to get $M\textbf v$ (they both use the scalars $a_i$). So summing up $a_i\textbf m_i$ is exactly how we can find the image of $\textbf v$ under transformation.

This is also the reason about "why the column space of $M$ is said to be the range of $M$", because any image under $M$ is a combination of $\{\textbf m_i\}$.

Answer 3

The important point to see the relation is that a matrix does not do arbitrary transformations on the vector, but linear transformations. Using this fact, the two viewpoints are easily mapped to each other.

For this behalf, let's give the two views of the matrix their own name, so we can more easily talk about them. The 3Blue1Brown view (respectively its generalization to arbitrary dimensions) is of the matrix as transformation $$\mu:\mathbb R^n\to\mathbb R^m, v\mapsto Mv$$ where $M$ is an $m\times n$ matrix (that is, $m$ rows, $n$ columns). That is, the transformation takes an input vector $v$ and turns it into an output vector $\mu(v)$. I'll call that the transformation viewpoint.

On the other hand, Gilbert Strang's view is of the matrix as a list of $n$ column vectors $c_j\in\mathbb R^m$, and the vector $v$ as a list of $n$ numbers $v_j$ (arranged in a column) that tells us how to combine the column vectors. I'll call that the column viewpoint.

So now let's start from the transformation viewpoint, and first remember that the standard basis in $\mathbb R^n$ consists of vectors $e_j$ which have an $1$ in row $j$ and a $0$ everywhere else, or short, $(e_j)_k=\delta_{jk}$

Now we can write each vector $v$ in this basis as linear combination $$\sum_{j=1}v_je_j$$ where due to the special form of the basis, the $v_j$ are exactly the entries of the column vector.

Next, recall the fact mentioned in the beginning, that the transformation $\mu$ is not arbitrary, but a linear transformation. That is, with scalars $a,b$ and vectors $u,v$ we have $$\mu(a u+b v)=a \mu(u)+b \mu(v).$$ Now let's apply this to the expansion of $v$ in the vector basis: $$\mu(v)=\mu(\sum_{i=1}^n v_je_j) = \sum_{i=1}^n v_j\,\mu(e_j)$$ Thus the result of the transformation is the linear combination of the $\mu(e_j)$ given by the vector components $v_j$.

But what are those $\mu(e_j)$? Well, rember the definition of $\mu$: $$(\mu(e_j))_l = (Me_j)_l = \sum_k M_{lk}\delta_{jk} = M_{lj} = (c_j)_l$$ In other words, the image of $e_j$ is simply $c_j$, the $j$-th column vector of M!

So in summary we get $$\mu(v) = \sum_{j=1}^n v_j c_j$$ where on the left you have the transformation viewpoint, and on the right you have the column viewpoint. Thus this equation tells you that both viewpoints are ultimately equivalent.

On the way, we have also seen that the column vectors are nothing else than the transformed standard basis vectors.

Now both views have their strength and weaknesses. The transformation view has the strength of being basis independent. In that sense it is the more geometric view, as the basis is something that is not inherent in the geometry, but a choice by you (the fact that you can choose a basis is of course inherent, as are certain properties of it, but the basis is not fixed by those properties). On the other hand, it de-emphasizes the linearity.

The column viewpoint, on the other hand, stresses the linearity, as it explicitly looks at the linear combinations of columns. On the other hand, it is basis dependent, and therefore de-emphasizes the geometric aspects of linear transformations.

Answer 4

As a starting point we can think to $\mathbb{R}$ as a vector space over $\mathbb{R}$ of dimension $1$. In this space the equation $ax=b$ is a linear equation that can be interpreted in different ways.

1) We can think at $a$ as a linear operator, so the equation asks for a vector (the number $x$) such that the linear transformation $ax$ is the given vector $b$.

2) we can think at $a$ as a vector (it is a real number) and the equation asks to find a coefficient $x$ such that this vector becomes the vector $b$, i.e. asks for the coefficient that gives $b$ as a linear combination of $a$.

In the $1$-dimension space $\mathbb{R}$ this seems a specious distinction, but it becomes evident in a space of larger dimension, where the linear equation becomes the linear system

$A\vec x=\vec b$

that can have the two interpretations that you have encountered.