I have two 2D unit vectors a and b. I'd like to find the rotation matrix that rotates a to b. The formulas I see online are for a rotation matrix are
$$ \left( \begin{matrix} \cos \theta & - \sin \theta \\ \sin \theta & \cos \theta \\ \end{matrix} \right) $$
And I can get the angle between a and b with
$$ \theta = \cos^{-1} (\mathbf{a} \cdot \mathbf{b}) $$
My problem is that that doesn't give me the direction. For example, if $\theta$ is $ \pi/2 $ when maybe the matrix should use $ -\pi/2 $
You don’t need to compute the angle explicitly, or indeed refer to an angle at all.
Observe that the result of rotating any vector $(x,y)^T$ 90 degrees counterclockwise is $(-y,x)^T$. Then, using the fact that the columns of a transformation matrix are the images of the basis vectors, the matrix $$\begin{bmatrix}x&-y\\y&x\end{bmatrix}$$ represents a rotation that maps $(1,0)^T$ onto the unit vector $(x,y)^T$. Therefore, a rotation that takes the unit vector $\mathbf a=(x_a,y_a)^T$ to the unit vector $\mathbf b=(x_b,y_b)^T$ has the matrix $$\begin{bmatrix}x_b&-y_b\\y_b&x_b\end{bmatrix} \begin{bmatrix}x_a&-y_a\\y_a&x_a\end{bmatrix}^{-1} = \begin{bmatrix}x_b&-y_b\\y_b&x_b\end{bmatrix} \begin{bmatrix}x_a&y_a\\-y_a&x_a\end{bmatrix} = \begin{bmatrix}x_ax_b+y_ay_b&x_by_a-x_ay_b \\ x_ay_b-x_by_a & x_ax_b+y_ay_b\end{bmatrix}.$$