Since we know that a group action $F: G\times S\rightarrow S$ can be described by a functor $F : G\to \mathrm {Set}$, See here. I would like to know how can we describe a topological group action with a functor.
I tried use the same method $F : G\to \mathrm{Top}$, however, this $F$ may not be consisted with the topological structure, that's to say, may not be a continuous map $G\times T\to T$. So, is there any methods for us to describe the 'continuty' of the functor $F$ ?
The situation is exactly as in the case of non-topological group actions, except we have to work internal to $\mathsf{Top}$.
First, a groupoid internal to $\mathsf{Top}$ is a pair of topological spaces $O$ and $A$ (for Objects and Arrows), plus continuous maps
If it's not obvious, it's worth working out why we need the domain for composition to be the pullback $A \times_O A$, and what the continuous maps are in the pullback.
These continuous maps are supposed to make certain natural diagrams commute, which I don't feel like writing down. They're exactly the usual axioms for a category, but expressed as commuting diagrams. What matters for us is that a topological group $G$ is the same data as a one-object groupoid internal to $\mathsf{Top}$. That is, an internal groupoid where $O = \{ \star \}$ is the terminal object!
Now we can also write down the definition of an "internal functor" from an internal category $\mathcal{C}$ to $\mathsf{Top}$. Again, I don't feel like saying exactly what this is, but I'll write it down in the special case that $\mathcal{C}$ is a one-object groupoid (which is the situation relevant for us).
An internal functor from an internal one-object groupoid $\mathcal{G}$ to $\mathsf{Top}$ is the data of a topological space $X$ (really with a map from $X \to \{ \star \}$, but there's only one choice since $\{ \star \}$ is terminal) and a continuous map $\alpha : G \times X \to X$ satisfying the axioms:
You surely recognize these as the usual axioms for a topological group action, so we see the (expected) punchline:
Which is entirely analogous to the case for $\mathsf{Set}$. You might expect that this continues to work for groups internal to other categories as well, and indeed it does. You might also want to read more about internal categories (especially to get some of the details that I've been too lazy to write down here). For information on these, you'll likely be interested in Chapter $8$ (especially Sections $8.1$ and $8.2$) of Borceux's Handbook of Categorical Algebra, Volume $1$.
I hope this helps ^_^