Question:
The resultant of two concurrent forces, $\vec{P}$ and $\vec{Q}$, $[P>Q]$, trisects the angle between them. Show that the angle between them is $3\cos^{-1}\left(\frac{P}{2Q}\right)$ and the magnitude of the resultant is equal to $\frac{P^2-Q^2}{Q}$.
My book's attempt:
$$R\cos(0^{\circ})=P\cos(-\alpha)+Q\cos(2\alpha)\tag{1}$$
$$[\text{Breaking the convention,}\ R\cos(0^{\circ})=P\cos(\alpha)+Q\cos(-2\alpha)]$$
$$R\sin(0^{\circ})=P\sin(-\alpha)+Q\sin(2\alpha)\tag{2}$$
$$[\text{Breaking the convention,}\ R\sin(0^{\circ})=P\sin(\alpha)+Q\sin(-2\alpha)]$$
Using $(2)$,
$$0=P\sin(\alpha)-Q\sin(2\alpha)$$
$$P\sin(\alpha)=Q\sin(2\alpha)$$
$$P\sin\alpha=2Q\sin\alpha\cos\alpha$$
$$\cos\alpha=\frac{P}{2Q}$$
$$\alpha=\cos^{-1}\left(\frac{P}{2Q}\right)$$
$$...$$
My comments:
In $(1)$ and $(2)$, why is $\alpha$ positive and $2\alpha$ negative? I know that going counterclockwise from the x-axis is positive and going clockwise is negative, but in $(1)$ and $(2)$, both seem to be going in the same direction with respect to $OA$, so why are their signs different?
My question:
- Why are the signs of $\alpha$ and $2\alpha$ different?

It is because in the solution provided in your textbook, OC is considered as the reference line instead of OA(which you have assumed in your comments).