Let $X$ be a topological space, $H$ be the group of homeomorphisms of $X$, and $T$ be a topological group acting on $X$ on the right. Let $\phi:T\rightarrow H$ be given by $t\mapsto \Pi^t$ where $\Pi^t:X\rightarrow X$ is given by $x\mapsto xt$.
According to my book, $\phi$ is a group homomorphism but it is really not unless the order of function composition is reversed. Moreover, according to the book, $\ker \phi$ is a closed subgroup of $T$ if $X$ is Hausdorff. How can we prove it?
You are correct that there is a problem with the order of composition. I will therefore write $T$ as having a left action, such that $\phi(st)=\phi(s)\phi(t)$.
I believe that putting the compact-open topology on $H$ is the 'better' argument, but it can be done by hand as well.
We will show that the complement of $\ker \phi$ is open. To do this we will find an open neighborhood for any $t\notin \ker\phi$ which is disjoint to $\ker \phi$.
Let $t\notin \ker \phi$. Then $\Pi_t\neq 1_X$. In particular there is $x\in X$ with $tx\neq x$. Because $X$ is Hausdorff there is a open subset $U\subset X$ with $tx\in U$ but $x\notin U$.
Consider now the evaluation at $x$, i.e. $ev_x:T\to X$ $ev_x(t)=tx$. It is continuous. Then clearly $t\in ev_x^{-1}(U)$ and $ev_x^{-1}(U)$ is disjoint from $\ker\phi$. Indeed if $s\in \ker \phi$, then $ev_x(s)=sx=x\notin U$.