On page 4032 of this paper on CAT scan there is a distance $d_{ij}$ calculated I presume in the direction of the axis $r$ with the following formula
... $d_{ij} = r_i - x_j \cos \theta_i - y_j \sin\theta_i$ is the signed distance between the projection strip $h_i$ and the $j$-th grid point $(x_j,y_j)$ as shown in Fig. 2.
What is the geometrical reasoning behind $d_{ij} = r_i - x_j \cos \theta_i - y_j \sin\theta_i$?
I'm asking because considering the rotation matrix
$$\begin{bmatrix} \cos\theta & - \sin \theta\\ \sin\theta& \cos \theta\end{bmatrix}$$
the $x_j$ coordinate would rotate to $x_j\cos\theta -y_j\sin\theta,$ and therefore, the distance would be
$$d_{ij}=r_i-x_j\cos\theta +y_j\sin\theta$$

This distance is measured parallel to $r$, which is perpendicular to $h_i$, so it’s just the difference between $r_i$, and the (signed) length of the orthogonal projection of $p_j=(x_j,y_j)$ onto $r$ a.k.a. the scalar projection of $p_j$ onto $r$. The unit vector in the direction of $r$ is $u=(\cos\theta_i,\sin\theta_i)$, so this projection is equal to $(x_j,y_j)\cdot(\cos\theta_i,\sin\theta_i)$, leading directly to the formula $$r_i-x_j\cos\theta_i-y_j\sin\theta_i.$$
You can also derive this formula via a rotation, as you attempted to do, but note that you need to rotate through an angle of $-\theta_i$ instead, which maps $(x_j,y_j)$ to $(x_j\cos\theta_i+y_j\sin\theta_i,-x_j\sin\theta_i+y_j\cos\theta_i)$.