Let $G$ be a finite abelian group with $x, y \in G$ such that $|x| > 2$, $|y|$ divides $|x|$ and $y \notin \langle x \rangle$. Then there exists no automorphism $f$ on $G$ such that $f(x) = x$, $f(y) = y$ and $f(xy) = (xy)^{-1}$.
My attempt: If there exists an automorphism $f$ on $G$ such that $f(x) = x$, $f(y) = y$ and $f(xy) = (xy)^{-1}$. Then $xy = x^{-1} y^{-1}$ gives $x^2 = y^{-2}$.
I am stuck here. Could anyone give me any idea how to go further? I am so thankful to you.
Given $f(x)=x$ and $f(y)=y$, then since $f$ is a group homomorphism you have $f(xy)=f(x)f(y)=xy$. Thus $f(xy)=(xy)^{-1}$ if and only if $xy$ has order dividing 2. Since $y\neq x^{-1}$ by your assumption, it's if and only if $xy$ has order exactly 2. Derek Holt's comment shows this is quite easy to happen:
The problem arises in this example because $\langle\,x\,\rangle\cap \langle\,y\,\rangle = \langle\,x^2\,\rangle\neq 1$. If you change the hypothesis "$y\not\in\langle\,x\,\rangle$" to $\langle\,x\,\rangle\cap \langle\,y\,\rangle=1$, or even $\langle\,x^2\,\rangle\cap \langle\,y^2\,\rangle=1$, then the result will carry through. For then $(xy)^2 = x^2 y^2 =1$ if and only if $y^2 =x^{-2}$. But since $x^2\neq 1$ by assumption this would imply $\langle\,x\,\rangle\cap \langle\,y\,\rangle\supseteq\langle\,x^2\,\rangle\cap \langle\,y^2\,\rangle\neq 1$.
Note that in either case no use of "$|y|$ divides $|x|$" is necessary.