The equations $$Ax+By+C=0$$ and $$x\cos \alpha + y\sin \alpha - p=0$$ will represent one and the same straight line if their corresponding coefficients are proportional: $$\dfrac {\cos \alpha}{A} = \dfrac {\sin \alpha}{B}=\dfrac {-p}{C}=k(\textrm {say})$$ So that: $$\cos \alpha=Ak$$ $$\sin \alpha =Bk$$ $$-p=Ck$$ Now, $$A^2k^2+B^2k^2=1$$
$$k^2=\dfrac {1}{A^2+B^2}$$ $$k=\pm \dfrac {1}{\sqrt {A^2+B^2}}$$
If $C>0$, then $$k=-\dfrac {1}{\sqrt {A^2+B^2}}$$
Here I don't understand the portion after "If $C>0$, then……
By convention, $p\ge 0$, since it is the distance from the origin to the line. (The normal vector to the line is $(\cos\alpha,\sin\alpha)$, and $(x,y)\cdot(\cos\alpha,\sin\alpha) = x\cos\alpha+y\sin\alpha$ gives this distance.) So if $k=-p/C$ and $p$ and $C$ are positive, then $k$ must be negative.