I am reading "Topics in Algebra 2nd Edition" by I. N. Herstein.
The author defined the second permutation representation of $G$ as follows:
Let us call the realization of a group $G$ as a set of permutations given in Problem 1, Section 2.9 the second permutation representation of $G$.
Problem 1, Section 2.9 on p.74:
Let $G$ be a group; consider the mappings of $G$ into itself, $\lambda_g$, defined for $g\in G$ by $x\lambda_g=gx$ for all $x\in G$. Prove that $\lambda_g$ is one-to-one and onto, and that $\lambda_{gh}=\lambda_h \lambda_g$.
Let $f:G\to G'$ be a mapping such that $f(xy)=f(x)f(y)$ holds for all $x,y\in G$.
Then, $f$ is called a homomorphism.
My question is here:
Let $g:G\to (G',\cdot)$ be a mapping such that $g(xy)=g(y)\cdot g(x)$ holds for all $x,y\in G$.
Then, can we call $g:G\to (G',\cdot)$ a homomorphism?
My observation:
Let $G'':=G'$.
And for all $x,y\in G''$, let $x*y:=y\cdot x$.
Then, $(G'',*)$ is a group.
Let $h:G\to (G'',*)$ be a mapping such that $h(x):=g(x)$ for all $x\in G$.
Then, $h:G\to (G'',*)$ is a mapping such that $h(xy)=g(xy)=g(y)\cdot g(x)=g(x)*g(y)=h(x)*h(y)$.
So, $h:G\to (G'',*)$ is a homomorphism.
No. Let $G=G'$ be any nonabelian group and let $g:G\to G$ be the map $x\mapsto x^{-1}$. Then $g$ is an antihomomorphism (the property you describe), but not a homomorphism (select any two elements of $G$ which do not commute). If you want an explicit example for which the homomorphism property does not hold, I would suggest taking $G=\operatorname{GL}_2(\mathbb{R})$, let $g$ be matrix inversion and consider the matrices $$\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix},\quad \begin{pmatrix} 1 & 0 \\ 1 & 1 \end{pmatrix}.$$
Your observations are unfortunately not very relevant here. You have only shown that $h:G\to G''$ (as you have defined) is a homomorphism if $g$ is an antihomomorphism.