Ok so for example the matrix in REF
\begin{bmatrix} 1 & -2 & 5 & 0 & 3 \\ 0 & 1 & 3 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 \end{bmatrix}
has the row space of \begin{array}{ccccccccccccccc} \{[1 & -2 & 5 & 0 & 3],[0 & 1 & 3 & 0 & 0].[0 & 0 & 0 & 1 & 0]\} \end{array}
which is said to have a dimension of 3. I understand that this is based on the number of matrices, but isn't a "dimension" of 3 supposed to be a 3x1 matrix?
I'm a bit confused as to why they refer to it as the dimmension of the row space.
The dimension of a vector space $V$ is the number of elements in any basis of $V$.
The row space of an $m\times n$ matrix $A$ is defined as $\operatorname{Row}(A)=\operatorname{Span}\{v_1,\dotsc,v_m\}$ where $v_1,\dotsc,v_m$ are the rows of $A$. Note that $\operatorname{Row}(A)$ is a subspace of $\Bbb R^n$.
Now, if $W$ is a subspace of $\Bbb R^n$ and $\beta=\{w_1,\dotsc,w_k\}$ is a basis for $W$, then $\dim W=k$.
In your case, you have a $4\times 5$ matrix $A$. Note that $\operatorname{Row}(A)$ is a subspace of $\Bbb R^5$ and $$ \beta=\{ \begin{bmatrix} 1&-2&5&0&3 \end{bmatrix}, \begin{bmatrix} 0&1&3&0&0 \end{bmatrix}, \begin{bmatrix} 0&0&0&1&0 \end{bmatrix} \} $$ is a basis for $\operatorname{Row}(A)$. Hence $\dim\operatorname{Row}(A)=3$.
In short, the dimension is a number, not a matrix.