My question is more general, but it is illustrated in this example:
I was trying to answer this question, which asks to solve for $xy$ in the equation $x^2 + xy + y^2 + \sqrt3 y + 1 = 0$. One idea that I had was to set $xy = c$, substitute $x = \dfrac cy$, and then solve for $y$ to get $y = ($some expression in terms of $c)$. Then I was going to substitute $y = \dfrac cx$ to get $x = ($ another expression in terms of $c)$. Lastly, I was going to set up the equation
$($some expression in terms of $c) \cdot ($another expression in terms of $c) = c$
How do I know if my last equation will just reduce to $c = c$ and all my work will have been for nothing?
More generally, how do I know when making substitutions will give me a new equation with new information, or just an identity?
From your first substitution you got $y=f(c)$. Then, when you plug in $y=\frac{c}{x}$ you only get $x=\frac{c}{f(c)}$, i.e. $c=\frac{c}{f(c)}f(c)=c$ so you're going in circles.
The problem is you're not utilizing any new information, but recycling what you already used. Basically, the only substitution made here was $xy=c$, rewritten the first time as $x=\frac{c}{y}$. The second substitution was again just a rewrite of this same equation, no additional information was given and this will typically get you going in circles.
It's a lot like when, for example, you have a linear system \begin{align} x-y&=2\\ x+y &=4 \end{align} You would get from the first equation that $x=2+y$, but you wouldn't plug that back into the same equation, but into the second one, which gives you one more piece of information.