What have Vectors and Matrices got to do with each other?

Question

In my undergraduate course work I learnt Vectors (as in those in vector space with magnitude and direction) separately from Matrices - an $n \times m$ array of numbers. However, after sitting in for a class on optimization theory the faculty member seems to use the term row/column vectors as if to refer to vectors in vector space.

I always assumed row/column vector to represent an 'array' of number arranged as such and not really as vectors in vector space and definitely not as a matrix of vectors. Is this interpretation common? Is this more like a dual interpretation? That any arbitrary matrix can be interpreted as vectors in vector space and can be operated upon using vector arithmetic and have a valid interpretation of the results of the operations? Or is the term just overloaded?

Am I missing something here or do I have it wrong or was I just missing some intuition :)

Answer 1

Yes, this is a very common interpretation of matrices. It is quite natural when you get used to it (in fact this is often taught as the "natural" way to think of matrices).

For example, multiplication of a matrix by a vector can be thought of as taking a linear combination of the matrix's columns. Multiplication of a matrix by a matrix results in all the dot products of the columns of the matrices.

This is especially common in applied mathematics, where data objects are represented as vectors that are multidimensional, and thus a set of data is represented as a matrix. The same goes for optimization, where many interesting problems typically involve optimizing over a set of vectors in a high-dimensional space. Often useful operations on the dataset can be translated into matrix transformations.

Answer 2

An "array" of $d$ numbers together with a suitable inner product constitutes a $d$-dimensional vector space. The classic example is the coordinate space $\mathbb{R}^n$ with the usual dot product. A "2d array" a.k.a. a matrix, can be interpreted as a linear operator in that space, for instance the rotation matrix.

You can further distinguish between column vectors - the members of the vector space, and row vectors, members of the dual space.

Also,you can look at the rows of the matrix as a vectors spanning a "row space" whose dimension is the dimension of the image space of the operator described by the matrix. This property is commonly used in computer science for image compression (lookup PCA).