In the following all variables are assumed to be integers.
It is easy to write a Diophantine equation which has solutions only when $N$ is a square. i.e.
$$N=A^2$$
It's trivial to write a Diophantine equation which has solutions if and only if $N$ is divisible by 4:
$$N = 4A$$
Also it is fairly easy to write a Diophantine which has solutions if and only if $N$ is not divisible by 4:
$$(N-4A-1)(N-4A-2)(N-4A-3)=0$$
But how about a Diophantine equation which has solutions if and only if $N$ is not a square number?
(sum, product and minus only can be used).
Here's one approach, though possibly not the simplest.
A number $N$ is not a perfect square if $A^2+1 \le N \le (A+1)^2-1$ for some $A$. But how can we encode $X \le Y$? Over the real numbers, the standard trick would be to write $Y = X + Z^2$, because $Z^2$ is always nonnegative. Over the integers, that doesn't quite work, but we know that $X \le Y$ if and only if there are four integers $P,Q,R,S$ such that $Y = X + P^2 + Q^2 + R^2 + S^2$, by using Lagrange's four-square theorem.
So we get $$ N = A^2+1 + B^2 + C^2 + D^2 + E^2 \text{ and } A^2+2A = N + F^2 + G^2 + H^2 + I^2 $$ but we probably want to write this as a single equation. To do this, just take $$ (N - A^2-1-B^2-C^2-D^2-E^2)^2 + (A^2+2A-N-F^2-G^2-H^2-I^2)^2 = 0. $$ Over the integers (or even the reals), $X^2+Y^2=0$ only if $X=Y=0$, giving us what we wanted.
Actually, the above only characterizes positive nonsquares: if $N$ is negative, there's no value of $A$ we can choose. But we can multiply the equation above by an equation representing $N\le-1$ easily, fixing this issue.