Let $G$ be a compact Lie group acting on a manifold $M$ such that the stabilizers $G_x$ are finite for all $x \in M$. (Recall that, $G_x = \{g \in G : g\cdot x = x\}$). I am trying to show that, such an action is proper.
Here's what I have so far: let $F : G \times M \to M \times M$ be the map $(g, x) \mapsto (g\cdot x, x)$. Given any compact set $K \subset M \times M$, we need to show that $F^{-1}(K)$ is compact. Let $K_1$ and $K_2$ be the projections of $K$ onto its components in $M$. Clearly, $K_1$, $K_2$ are compact, and hence so is $N = K_1 \cup K_2$. Also, $K \subset N \times N$. Now, it suffices to show that $F^{-1}(N\times N)$ is compact (since $F^{-1}(K)$ is a closed subset of $F^{-1}(N \times N)$).
Now, $F^{-1}(N \times N) = \{ (g, x) : g\cdot x \in N, x \in N\} \subset \{g \in G : g\cdot N \cap N \ne \varnothing\} \times N$. If we can show that $\{g \in G : g\cdot N \cap N \ne \varnothing\}$ is a closed subset of $G$, then we will be done (since $G$ is compact). This is where I am stuck, I can't show why this subset of $G$ is closed.
So my questions are: is this even the correct approach? (because, it can very well happen that the subset of $G$ that I described above may not be closed, but the original statement still holds)? But, if this is indeed the correct approach how do I show that that set is closed.
Lastly, does this generalize to topological groups? (If we take $G$ to be a compact topological group and $M$ to be a topological space, does the similar statement still hold?)
Your method seems fine to me.
In order to prove this set is closed, suppose $g_i\in \{g|gN\cap N\neq \varnothing\}$ for $i\in\mathbb{N}$ and that $g_i\longrightarrow g_0$, and let us prove $g_0 N \cap N\neq \varnothing$. By definition, for each $i$ there is $n_i\in N$ such that $g_in_i\in N$. We can take a convergent subsequence $n_{i_j}\longrightarrow n_0$, where $n_0\in N$, and certainly $g_{i_j}\longrightarrow g_0$. Therefore $(g_{i_j},n_{i_j})\longrightarrow (g_0,n_0)$, so by continuity of the action and closedness of $N$ we get that $g_0 n_0\in N$. Therefore, $g_0\in \{g|gN\cap N\neq \varnothing\}$, as needed.
The statement remains true if we only assume $G$ is a compact (Hausdorff) topological group and $M$ is a Hausdorff topological space. The same proof works, you just need to replace "sequence" with "net" (and "subsequence" with "subnet").
Observe that since in your case $M$ is locally compact (and therefore so is $M\times M$), the map of the action is proper if and only if it is closed and has compact fibres. Proving that it is closed should be easy, and proving that it has compact fibres is a little easier than proving the preimage of a compact set is compact. (It's about the same, only with less technicalities).
I hope this helps, let me know if something isn't clear.