I start with a simple problem that I was able to solve: Let $G$ be a group. Let $a\in G$. Assume that $H := \{g \in G : g^2 \neq a\}$ is a subgroup of $G$. The question: Can we define $H$ with a "positive" formula, not involving the symbol $\neq$? The answer in this case is positive. Most of the time $H=G$, and in this case $H$ is defined by the positive formula $1=1$, but other times $H = \{g \in G : g^2 = 1\}$.
My intuition is that subgroups defined with a formula should look like closed subsets of a topology and therefore they should be defined with a "positive" formula.
For $a, b \in G$, one could look at subgroups defined by the equation $gag \neq b$, or more generally by the equation $w(g)\neq 1$ for some word $w(x)$. I believe all such subgroups should be defined by positive formulas. Any ideas or counter-examples or any known researches?
The question is of course interesting only for infinite subgroups.
Let me explain the solution to two special cases of the equational/co-equational problem.
Special subcase 1: the restriction of the problem to abelian groups only, and
Special subcase 2: the restriction of the problem to the special case mentioned in the problem statement. That is, $H = \{g\in G\;:\;g^2\neq a\}$. Here I do not assume that the group is abelian.
Solution to special subcase 1:
Assume that $G$ is abelian and that $H=\{g\in G\;:\;s(g)\neq t(g)\}$ for some $1$-variable equation $s(x)=t(x)$, possibly with parameters. Moving the variables to the left of the equals sign and the parameters to the right, the equation $s(x)=t(x)$ reduces to something of the form $x^n=a$. Hence the co-equational definition of $H$ is equivalent to $H=\{g\in G\;:\;x^n\neq a\}$. (Note that $1\in H$, so $1^n\neq a$, so $a\neq 1$.) I claim that either (i) $H=G$ and $H$ is defined equationally with the equation $x=x$ [$H=\{g\in G\;:\;g = g\}$] or (ii) $H\neq G$ and $H$ is defined equationally with the equation $x^n=1$ [$H=\{g\in G\;:\;g^n = 1\}$].
To show this, assume that $H\neq G$ and that $p\in G-H$. By the co-equational definition of $H$, $p^n=a\neq 1$. Let $W=\{g\in G\;:\;g^n=1\}$. Now observe that if $h\in H$, then $ph\notin H$, so $a = (ph)^n = p^nh^n = ah^n$. Canceling $a$ yields $h^n=1$. This argument shows that $H\subseteq W$. Conversely, choose any $g\in W$. Since $g^n=1\neq a$, then $g$ satisfies the co-equational definition of $H$, so $g\in H$. Altogether this shows that $H=W$, which means that $H$ is equationally defined by $x^n=1$. $\Box$
Solution to special subcase 2:
I do not assume that I am working with abelian groups, but I do assume that $H$ is defined co-equationally by $H=\{g\in G\;:\;g^2\neq a\}$. Since $1\in H$, we have $1^2\neq a$, so $a\neq 1$. This implies that $a^2\neq a$, so we have $a\in H$ by the co-equational definition of $H$. Also, from Kenta S's comment on the original question, we have $a^2=1$. Note that this implies that any $p\in G-H$ has order $4$, since $p^4=a^2=1$, but $p^2=a\neq 1$.
I claim that either (i) $H=G$ and $H$ is defined equationally with the equation $x=x$, [$H=\{g\in G\;:\;g = g\}$] or (ii) $H\neq G$ and $H$ is defined equationally with the equation $x^2=1$ [$H=\{g\in G\;:\;g^2 = 1\}$], or (iii) $H\neq G$ and $H$ is defined equationally with the equation $p^{-1}xp=x^{-1}$ for some parameter $p\in G-H$ [$H=\{g\in G\;:\;p^{-1}gp = g^{-1}\}$].
Preliminary observation.
If $h\in H$ and $p\in G-H$, then $p^{-1}hp=h^{-1}$. ($p$ conjugates any element of $H$ to its inverse.)
To see this, note that $ph\in G-H$, so $(ph)^2=a=p^2$. The equation $phph=pp$ may be rewritten as $(pp)^{-1}phph=1$ and then simplified to $p^{-1}hp=h^{-1}$, which shows that every element of $H$ satisfies the equation $p^{-1}xp=x^{-1}$.
Case 1. $[G:H]>2$.
In this case I claim that the equationally-defined subset $X=\{g\in G\;|\;g^2=1\}$ is exactly $H$. First, $X\subseteq H$ since $X$ consists of elements of exponent $2$, while $G-H$ consists of elements of order $4$. Conversely, assume that $h\in H$ and that $H, pH, qH$ are distinct cosets of $H$. Then $p, q, pq\in G-H$ while $h\in H$, so each of $p, q, pq$ conjugates $h$ to its inverse. This means that $h^{-1}=(pq)^{-1}h(pq)=q^{-1}(p^{-1}hp)q=q^{-1}h^{-1}q=h$, so $h^2=1$ and $h\in X$. This proves that $H = X$, completing the proof for Case 1.
Case 2. $[G:H]=2$.
In this case I claim that for any fixed $p\in G-H$ the equationally-defined subset $Y=\{g\in G\;|\;p^{-1}gp=g^{-1}\}$ is exactly $H$. We have already shown that $H\subseteq Y$ in the Preliminary Observation, so let's argue that $Y\subseteq H$. This is equivalent to showing that $Y\cap (G-H)=\emptyset$. To obtain a contradiction, assume that there is some $q\in Y\cap (G-H)$. Since $q\in Y$ and $p\in G-H$, the Preliminary Observation guarantees that $p^{-1}qp=q^{-1}$. Right multiplying by $q$ and left multiplying by $p^2$ yields $pqpq=p^2=a$, so $(pq)^2=a$. This means that $pq$ fails the co-equational definition of $H$, so $pq\in G-H$. But this contradicts the assumption that $[G:H]=2$, as I now explain. Under the assumption $[G:H]=2$, $H\lhd G$ and $G/H$ is a $2$-element group. For each of $p, q, pq\in H$ we have that $pH = qH= pqH$ represent the nonidentity element of this quotient group. By the multiplication table of the $2$-element group, we should have $pH\cdot qH = H$, but we have $pH\cdot qH=pqH\neq H$. This contradiction shows that $Y\cap (G-H)=\emptyset$, so $Y\subseteq H$, so $Y=H$. This completes the proof of special subcase 2. $\Box$
Finally, note that if $G=Q_8$ is the $8$-element quaternion group and $a$ is the nonidentity central element, then the center $Z(G) = \{g\in G\;:\;g^2\neq a\}$ is a co-equationally defined subgroup, and it is equationally defined by $\{g\in G\;|\;g^2=1\}$ (the Case 1 equational definition). But $Z(G)$ is NOT equationally defined by $\{g\in Q_8\;|\;p^{-1}gp=g^{-1}\}$ for any $p\in Q_8$ (the Case 2 equational definition). On the other hand, if $G=\textrm{Dic}_3$ is the $12$-element dicyclic group and $a$ is the nonidentity central element, then the index-$2$ subgroup is defined co-equationally by $\{g\in G\;:\;g^2\neq a\}$. It is equationally defined by $\{g\in G\;|\;p^{-1}gp=g^{-1}\}$ (Case 2 equational definition) using any parameter $p$ of order $4$, but NOT by $\{g\in G\;|\;g^2=1\}$ (the Case 1 equational definition). Thus, when $H\neq G$, it seems that you really need to consider two cases.