In particular, I'm struggling with this question:
Give me a rotational matrix, a scaling matrix, or a reflection matrix and I can provide it quite easily. No online resources I've found give me much information on how to find a 3D shear matrix. I even tried asking this question on Chegg and I got a dead wrong answer. Because one thing I do KNOW about shear matrices from 2D is that they preserve area and that can be applied to any dimension so in $R^3$ the volume is preserved. And the "expert" on Chegg said the volume goes to 0, which is obviously not true!
So I'm hoping I'll find better luck asking it here. Thanks in advance!
A true shear matrix, according to Wikipedia, is an elementary matrix, and with only one non-zero off-diagonal entry, this cannot be done.
However, shear transformations are a bit more flexible. Specifically because they are not bound to shearing along the coordinate axes.
Note that there are infinitely many shear transformations $S$ for which $$ S\begin{bmatrix}a\\b\\c\end{bmatrix} = \begin{bmatrix}0\\0\\c\end{bmatrix}\tag{*} $$ This is possibly what gives you trouble finding a solution; if you try to solve a problem algebraically and you don't know that there are infinitely many solutions, it's going to look like you're missing an equation somewhere. So, with that said, let's find one such transformation and the matrix representing it.
Firstly, I'm going to assume that $c \neq 0$. That's because shears are invertible, and if $c = 0$, then by $(\text *)$ we must also have $a = b = 0$, and this becomes quite a boring exercise.
Secondly, we clearly have $$ S\begin{bmatrix}a\\b\\0\end{bmatrix} = \begin{bmatrix}a\\b\\0\end{bmatrix} $$ but I will assume that $$ S\begin{bmatrix}x\\y\\0\end{bmatrix} = \begin{bmatrix}x\\y\\0\end{bmatrix} $$ for all $x, y$. This is where the multiple solutions come in. If you make a basis $(u, v)$ for the $xy$-plane where $u = [a, b, 0]^T$, then $Su = u$, but we know very little about $Sv$. I assume here that $Sv = v$ as well, but we have no indication in the problem text whether this is so. $Sv$ doesn't even have to be in the $xy$-plane.
So, with this, we are ready to begin making a matrix of $S$: $$ S = \begin{bmatrix}1&0&\\0&1&\\0&0&1\end{bmatrix} $$ And I believe you can fill in the two missing numbers by using $(\text *)$.
Note that expressing this matrix in a basis where $[a, b, 0]^T$ and $[0, 0, 1]^T$ (or constant multiples of these) are two of the basis vectors, this transformation $S$ will have the elementary form required in the Wikipedia article on shear matrices.