An easy way to solve this problem. pcosα=425cos(30)
For minimum value of $P \cos\alpha$ should be $1$.Then $\alpha=0$.
But I want to solve this problem by using calculus.
Horizontal component:
$P\cos\alpha=425\cos(30)$
Vertical component
$P\sin \alpha+425\sin(30)=R$
How can I relate these two equations for differentiation?
What you wrote is not a horizontal component, but rather a criterion for vanishing horizontal component.
An actual horizontal component is $H=P\cos\alpha-425\cos(30^\circ)$.
And you need $H=0$, which implies: $$P\cos\alpha=425\cos(30^\circ)$$
Now you can solve it for $P$ with respect to unknown variable angle: $$P=\frac{425\cos(30^\circ)}{\cos\alpha}$$
And you want to find a minimum $P$ satisfying the above requirement, hence you need to minimize the above $P(\alpha)$. Determine $\frac{dP}{d\alpha}$ and find its zero. (We already know it should be $\alpha_0=0$.)
Then check for positive second derivative $\frac{d^2P}{d\alpha^2}|_{\alpha=\alpha_0} > 0$ to make sure you actually found a minimum.