Suppose $P(\cos(x), \sin(x))$ is a polynomial in $\mathbb R[X, Y]$ with degree $n$. I want to determine the number of roots of $P(\cos(x), \sin(x))$. Here I mean the possible values of the pair $(\cos(x), \sin(x))$, not roots in $x$. We exclude the case $P = \cos^2(x) + \sin^2(x) - 1$ and the ideal generated by $P$. Not sure whether this is enough but I meant to ask when we don't have this trivial case.
If we introduce the change of variable $$ \cos (x) = \frac{ 1-t^2}{1+t^2}, \sin(x) = \frac{2t}{1+t^2}.$$ Then $P(\cos(x), \sin(x))$ can be considered as a rational function with numerator $n(t) \in \mathbb R[t]$ having degree at most $2n$. But this would imply we could have at most $2n$ solutions for $t$. Thus at most $2n$ solutions for the pair $(\cos(x), \sin(x))$. Is this correct? However, I had a feeling that the number should be $n$.
That's basically right. You have to be a little careful in that $(\cos x,\sin x) =(-1,0)$ isn't included in your parametrisation, but the conclusion still holds.
You do need $2n$, not $n$. You can see this even when $n=1$. For instance $\cos x=\sin x$ has two solutions $(\cos x,\sin x)=(1/\sqrt2,1/\sqrt2)$ and $(-1/\sqrt2,-1/\sqrt2)$.