New to linear algebra. I'm reading Mike X Cohens book on Linear Algebra, "Linear Algebra: Theory, Intuition, Code".
I don't follow the logic behind the left inverse. I will try to explain what I mean using matrix T below, a tall matrix with dimensions 3*2 (p.346 in the book)
I transpose T and matrix multiply it by itself, returns a square matrix 2*2
I get the invers of that square matrix above by:
- compute the determinant,
- swap diagonal elements,
- multiply off-diagonal elements by -1 and
- divide by the determinant.
- So far so good, but here comes the part I don't understand. To get the left inverse, I should multiply the inverse of the square matrix by T transposed.
In my head, this is not valid matrix multiplication since the inverse of the square matrix is 22 and the T transpose is 23 - inner dimensions dosen't match?
- The resulting matrix, the left inverse
1.
$$T= \begin{bmatrix} 1 & 2 \\ 1 & 3 \\ 1 & 4 \\ \end{bmatrix} $$
$$T*T^T= \begin{bmatrix} 3 & 9 \\ 9 & 29 \\ \end{bmatrix} $$
2.
$(T^T*T)^{-1}$ = 1/6 * [[29,-9],[-9,3]]
3.
$(T^T*T)^-1$ * $T^T$ = $T^{-L}$
4.
$T^{-L}$ = $(T^T*T)^{-1}$ * $T^T$ = 1/6 [ [11,-3],[2,0], [-7,3] ]
$T^{-L}$*T = 1/6 [ [6,0], [0,6] ] # Returns the identity matrix
I would appricate if someone could explain why it's valid matrix multiplication and perhaps write it out so I grasp this step in the algorithm.
Let's look at the theory before jumping into the particular case. If $A$ is an $m \times n$ matrix where $m \ne n$ then its left inverse is a matrix, say $A_L$, such that \begin{equation}\tag{1} A_L A = I_n, \end{equation} where $I_n$ is an $n \times n$ identity matrix. Likewise, $A_R$ is its right inverse if \begin{equation}\tag{2} AA_R = I_m, \end{equation} where $I_m$ is an $m \times m$ identity matrix. Before proceeding kindly convince yourself that the dimensions of the identity matrices in equations (1) and (2) are correct.
Let us now try to find what $A_L$ is. Since $A$ is not a square matrix, it does not have the usual inverse. We can form a square matrix from $A$ in two ways. $A^TA$ will give an $n \times n$ matrix while $AA^T$ will result in an $m \times m$ matrix. If $AA^T$ is not singular, then its inverse will exist and \begin{equation}\tag{3} AA^T (AA^T)^{-1} = I_m \Rightarrow A (A^T (AA^T)^{-1}) = I_m. \end{equation} Comparing with equation (2) we get \begin{equation}\tag{4} A_R = A^T (AA^T)^{-1}. \end{equation} Likewise, if $A^TA$ is not singular, $(A^TA)^{-1}$ will exist and \begin{equation}\tag{5} (A^T A)^{-1}A^TA = I_n. \end{equation} Comparing with equation (1) we get \begin{equation}\tag{6} A_L = (A^T A)^{-1}A. \end{equation} In your case, \begin{equation}\tag{7} A = \begin{bmatrix}1 & 1 & 1 \\ 2 & 3 & 4\end{bmatrix} \end{equation} so that \begin{equation}\tag{8} AA^T = \begin{bmatrix}3 & 9 \\ 9 & 29\end{bmatrix} \end{equation}, \begin{equation}\tag{9} (AA^T)^{-1} = \frac{1}{6}\begin{bmatrix}29 & -9 \\ -9 & 3\end{bmatrix} \end{equation} and hence \begin{equation}\tag{10} A_R = \frac{1}{6}\begin{bmatrix}11 & -3 \\ 2 & 0 \\ -7 & 3 \end{bmatrix}. \end{equation} We now check, \begin{equation}\tag{11} AA_R = I_2. \end{equation} You can compute $A_L$ using equation (1).