When I first learned about 2d rotation matrices I read that you represented your point in your new coordinate system. That is you take the dot product of your vector in its current coordinate system and against the new i and j vector in the rotated coordinate system.
$\begin{bmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{bmatrix}p=p^\prime$
The columns represent the new i and j vectors.
I'm reading a book and it says,
there can be confusion because their are two forms of the equation. One is through the transformation of the component of p (always with respect to x,y), x,y into x',y' and the other is through the transformation of the unit vectors i,j into i',j'.
He then shows the new equation
$\begin{bmatrix}i^\prime\\j^\prime\end{bmatrix} = \begin{bmatrix}\cos\theta&\sin\theta\\-\sin\theta&\cos\theta\end{bmatrix}\begin{bmatrix}i\\j\end{bmatrix}$
Anyway, my question is wouldn't you use the same matrix to change the vector i to i'? I don't see why the matrix would be the transpose. Additionally, if the vector i is on top of j then the vector is 4x1 rows by columns and you can't multiply 2x2 * 4x1. I'm quite confused.
To begin with, in the second equation you shouldn't think of the vector $\mathbf{i}$ as $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$; as you say, that wouldn't make sense. That equation is just a symbolic way of writing the vector equations $\mathbf{i}' = (\cos\theta) \mathbf{i} + (\sin\theta) \mathbf{j}$ and $\mathbf{j}' = (-\sin\theta) \mathbf{i} + (\cos\theta) \mathbf{j}$, which say that the new basis $(\mathbf{i}',\mathbf{j}')$ is rotated by an angle $\theta$ (counterclockwise) relative to the old basis $(\mathbf{i},\mathbf{j})$.
As you'll see below, it's better to write it like this instead: $$ \begin{bmatrix} \mathbf{i}' & \mathbf{j}' \end{bmatrix} = \begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix} \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}. $$
Now for any vector $\mathbf{u}$, we can look at its coordinates with respect to the old basis, $\mathbf{u} = x \mathbf{i} + y \mathbf{j}$, or with respect to the new basis, $\mathbf{u} = x' \mathbf{i}' + y' \mathbf{j}'$. (Note: No prime on $\mathbf{u}$ in the second formula, since it's the same geometrical vector as in the first formula.) On matrix form, this can symbolically be written as $$ \mathbf{u} = \begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} \mathbf{i}' & \mathbf{j}' \end{bmatrix} \begin{bmatrix} x' \\ y' \end{bmatrix}. $$ Here we replace the primed basis with the expression given by the formula above; this results in $$ \begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix} \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix} x' \\ y' \end{bmatrix}, $$ hence (since coordinates with respect to a basis are unique) $$ \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix} x' \\ y' \end{bmatrix}, $$ which is the relation between the coordinates of the vector $\mathbf{u}$ in the old and in the new basis, just as your textbook says.
As you yourself can testify, your book is also correct in its statement "there can be confusion"! ;-)