I'm reading Chapter 3 from the book "The theory of finite groups" by Kurzweil and Stellmacher where they say the following:
Let $G$ act on a set $\Omega$. That is, for each $x\in G$ and $\alpha\in\Omega$, there exists an element $\alpha^x\in \Omega$ which satisfies $\alpha^1=\alpha$ and $\alpha^{xy}=(\alpha^x)^y$.
Define the mapping $\pi:G\to S_{\Omega}$ as $\pi(x)=x^\pi$ where $x^\pi: \alpha\mapsto \alpha^x$. Then $\pi$ is a homomorphism.
I'm having trouble justifying this claim because I'm only able to show that $\pi(xy)=\pi(y)\pi(x)$.
My attempt: Since $$\begin{align} (xy)^\pi(\alpha)&=\alpha^{xy}\\ &=(\alpha^x)^y\\ &=y^\pi(\alpha^x)\\ &=y^\pi(x^{\pi}(\alpha)), \end{align}$$ so $$\pi(xy)=\pi(y)\pi(x).$$
Where am I going wrong?
Yep, right actions and writing functions on the left gives you a parity issue. You can either write your functions on the right (which is super weird, my brain will never be comfortable with $(x)f$), or you can work with left actions, or you can throw an inverse in there, i.e., define $\pi$ by $x^\pi\colon\alpha \mapsto \alpha^{x^{-1}}$.
Not having read that particular book I'm not sure which of these three solutions the authors had in mind.
Edit: As you've said that the book writes functions as $x^f$, that means they're evaluating functions from left to right. So in the composition $fg$ the function $f$ is evaluated first: $x^{fg} = (x^f)^g$. Note this is backwards in order to how we normally write functions as evaluating from right to left: $(fg)(x) = f(g(x))$. Getting that order wrong is why you are getting an anti-automorphism.
So the solution is you should write your functions on the right. That means you need to prove that $(\alpha)(xy)^\pi = ((\alpha)x^\pi)y^\pi$.