Let $T : R^3 \to R^2$ be defined by $T(a, b, c) = (a+b, 2a-c)$. Determine $T^{-1} (1, 11)$. (Exercise 3.3.6 from Linear Algebra by Friedberg)
The solution manual solve a system of linear equation $ a + b =1 $ and $ 2a - c = 11$, and obtain $T^{-1} (1, 11) = \{(a, 1-a, 2a - 11) : a \in \mathbb{R}\}$.
But, my question is whether $T$ is invertible or not. I have a theorem that $T$ is invertible iff $[T]_{\beta}^{\gamma}$ is invertible, where $\beta$ and $\gamma$ are standard ordered basis for $\mathbb{R}^3$ and $\mathbb{R}^2$. Here, $[T]_{\beta}^{\gamma}$ is $2$ by $3$ matrix, which implies $T$ is not invertible. Am I missing something or is this question wrong?
Your thinking is correct; only square matrices can have an inverse in the usual sense of one-to-one mapping. The fact that $T^{-1}(1,11)$ is an infinite set also means that $T$ is non-invertible.
You can still write $T^{-1}$, but this is the preimage function that takes a codomain vector and returns all its preimages in the domain. If $T$ is an invertible transformation, $T^{-1}$ will be a proper one-vector-to-one-vector function.