I have this problem: Prove that $\phi\colon D_8\to(\mathbb Z/5\mathbb Z)^\times$ is not a homomorphism with $\phi(x) = \bar2$.
My prove is as follow, assume $\phi$ is a homomorphism, and we have $$rs = sr^3$$ Therefore, $\phi(rs) = \phi(sr^3) \iff \phi(r)\phi(s) = \phi(s)\phi(r^3) \iff \bar2 = \bar3$, which is not valid.
So $\phi$ is not a homomorphism. I'm not sure if I got it right because I cancelled $\phi(s)$ from both side.
I would appreciate any help.
Your answer is mostly right, but you have missed a step. Here's the reasoning:
Let $\phi\colon D_8\to\left(\mathbb Z/5\mathbb Z\right)^\times$ by such that $\phi\left(x\right)=\bar 2$. $D_8$ is generated by two elements $r,s$ subject to the relations $rs=sr^3,r^4=1,s^2=1$. Assume $\phi$ is a homomorphism. Then we obtain $$ rs=sr^3\implies\phi\left(rs\right)=\phi\left(sr^3\right)\\ \implies\phi\left(r\right)\phi\left(s\right)=\phi\left(s\right)\phi\left(r\right)^3 $$ Since $\left(\mathbb Z\backslash 5\mathbb Z\right)^\times$ is abelian, $\phi\left(r\right)\phi\left(s\right)=\phi\left(s\right)\phi\left(r\right)$. Thus, we obtain $$ \phi\left(s\right)\phi\left(r\right)=\phi\left(s\right)\phi\left(r\right)^3\implies\phi\left(r\right)=\phi\left(r\right)^3 $$ Where we have multiplied by $\phi\left(s\right)^{-1}$ on both sides (which exists as $\left(\mathbb Z\backslash 5\mathbb Z\right)^\times$ is a group). If $\phi\left(r\right)=\bar 2$, then this given us $\bar 2=\bar 3$, which is a contradiction. Thus $\phi$ cannot be a homomorphism.
Remark 1: However, there is a much slicker way to see this. Let $G,H$ be groups and $\phi\colon G\to H$ be a function such that $\phi\left(g\right)=c$ for some constant $c\in H$ which is not the identity of $H$. Then $\phi$ is not a group homomorphism, because $\phi$ does not take the identity of $G$ to the identity of $H$ and a group homomorphism always preserves the identity.
The advantage of this proof is that it works for arbitrary groups.
Remark 2: Also, your proof generalises to a stronger statement. Consider a group homomorphism $\phi\colon D_{2n}\to A$ where $A$ is some abelian group. $D_{2n}$ is generated by two elements $r,s$ subject to the relations $$ r^n=s^2=1\\ rs=sr^{n-1} $$ Applying $\phi$ to these relations we obtain $\phi\left(s\right)^2=1, \phi\left(r\right)^n=1$ $$ \phi\left(r\right)\phi\left(s\right)=\phi\left(s\right)\phi\left(r\right)^{n-1}\implies\phi\left(s\right)\phi\left(r\right)=\phi\left(s\right)\phi\left(r\right)^{n-1}\\ \implies\phi\left(r\right)=\phi\left(r\right)^{n-1}\implies\phi\left(r\right)^{n-2}=1 $$ Where the first implication used the fact that $A$ is abelian, in the second we multiplied by $\phi\left(s\right)^{-1}$ and in the third, we multiplied by $\phi\left(r\right)^{-1}$. Putting the facts that $\phi\left(r\right)^n=1$ and $\phi\left(r\right)^{n-2}=1$, we get that $\phi\left(r\right)^2=1$. So we now have the equations that $$ \phi\left(r\right)^n=\phi\left(r\right)^2=1 $$ If $n$ is even, then $n=2k$ and since $\phi\left(r\right)^n=\left(\phi\left(r\right)^{2}\right)^k=1^k=1$, the first equation becomes redundant.
On the other hand if $n$ is odd, then $n=2k+1$, and $1=\phi\left(r\right)^n=\phi\left(r\right)^{2k}\phi\left(r\right)=1^k\phi\left(r\right)=\phi\left(r\right)$.
Thus for any homomorphism $\phi\colon D_{2n}\to A$ (for any abelian group $A$) we have that
In the given case of $D_8$, we have $n=4$ and $\phi\left(x\right)=\bar 2$. This obviously doesn't satisfy the above statement as $\bar 2^2=\bar 4\neq\bar 1$