I have to show the following:
Let $G$ be a finite group of order $n$ and let $\phi$ be the Cayley (right) action $\phi: G \hookrightarrow S_G \cong S_n$ defined by $\phi(g)=[x \mapsto xg \quad \forall x \in G]$. Then if $g$ is an element of $G$ of order $2$, $\phi(g) \in A_n \iff 4\mid n$.
First thing: for any $g \in G$, $\phi(g)$, being an element of $S_n$, can be decomposed uniquely (up to order of various factors) into a product of disjoint cycles. I claim that if $o(g)=2$ only $2$-cycles can appear in the cycle decomposition of $\phi(g)$. This is due to the fact that $[\phi(g) \circ \phi(g)](x)=[\phi(g^2)](x)=x$ for all $x \in X$; in other words, if $ \phi(g)= \sigma_1 \cdot \ldots \cdot \sigma_m $ is the cycle decomposition of $\phi(g)$ and $x \in \mathrm{spt}(\sigma_i)$ for some $i$, then $\mathrm{spt}(\sigma_i)=\{x, xg\}$.
Besides, every $x \in G$ is moved by $g$, meaning that $[\phi(g)](x) =xg \ne x$ for all $x \in G$ since $g \ne 1_G$. Hence every $x \in G$ must belong to $\mathrm{spt}(\sigma_i)$ for some $i$. Thus, the problem of determining wheter $\phi(g) \in A_n$ for an involution $g$ is equivalent to the problem of deciding wheter the elements of $G$ can be rearranged in an even number of disjoint pairs. This can be done iff $4\mid n$.
I don't know if this argument really works; it seems a little bit "brute force", I have tried with some counting argument (with orbits and stabilizers) but I have gone nowhere. Is my proof correct? Can it be made more precise?
$$xg=x\implies g=1_G$$ Therefore, $\text{fix}(\phi(g))=\emptyset$ because $\text{ord}(g)=2$. Consequently, there are exactly $\frac{n}{2}$ orbits of $\phi(g)$ because from Ruffini's Theorem each orbit has size $2$. Therefore, $\phi(g)$ is the composition of $\frac{n}{2}$ transpositions. $$\therefore\;\phi(g)\in A_n\iff 4\vert n$$