Let $X, Y$ be two topological spaces. If group $G$ acts on $X$ and $f:X\to Y$ is a $G$-invariant continuous map,i.e., $f\circ g=f$, why does it induce a homeomorphism between the quotient space of $X$ by $G$ and $Y$, that is,
$$X/G\cong Y ?$$
Could you give me some help with details？ Thanks a lot.
This obviously cannot be true when $f$ is not surjective. But even if it is, it still isn't true.
Consider the singleton $Y=\{*\}$. Then any $f:X\to Y$ (and by "any" I mean the single possible choice) is $G$-invariant, regardless of what $X$, $G$ and the action are. And so, if $G$ doesn't act transitively on $X$ (i.e. $X/G$ has at least two elements), then $X/G$ cannot be homeomorphic to $Y$.
Another counterexample is $Y=X/G$ but we put a coarser, non-homeomorphic topology on $Y$. In particular if $X/G$ is not anti-discrete, then we can consider $Y=X/G$ with antidiscrete topology. Then the projection $f:X\to Y$ provides a counterexample.
You've mentioned in the comments orbifolds. In orbifolds it is assumed that the induced map $$X/G\to Y$$ $$[x]\mapsto f(x)$$ is a homeomorphism. It is an additional condition and it doesn't follow from other conditions.