Let $S$ is a set of all subgroups of group $G$ and for each element $a\in G$ we match a function $c_a: S \rightarrow S$: $$c_a(H) = aHa^{-1}$$ How to prove that we set an action from $G$ onto $S$.
I know that i have to prove that $c_a\in Perm(S)$ and $a\mapsto c_a$ is a homomorphism, but i don't how to do it.
On
Recall that a group action of $G$ on $S$ is as follows: $1 \cdot s = s$ for all $s \in S$, and for all $a,b \in G$ and $s \in S$ we have $a \cdot (b \cdot s) = (ab) \cdot s$. Proceeding, we see $1 \cdot H = 1 H 1^{-1} = H$, so the identity property of group actions is satisfied. Now let $a,b \in G$. Then $a \cdot (b \cdot H) = a \cdot (b H b^{-1}) = a(b H b^{-1})a^{-1} = ab H b^{-1}a^{-1} = ab H (ab)^{-1} = (ab) \cdot H$.
Another way to prove that $G$ acts on the given set is to prove that $c_e(H) = H$ for all $H\le G$, but this is trivially true and $c_{ab}(H) = c_a(c_b(H))$ for all $H \le G$. Using the definition of $c_a$ we have that:
$$c_{ab}(H) = (ab)H(ab)^{-1} = (ab)H(b^{-1}a^{-1})$$ $$c_a(c_b(H)) = c_a(bHb^{-1}) = (ab)H(b^{-1}a^{-1})$$
Therefore we can conclude that $G$ acts on the set of subgroups of $G$.
On the other side using your method all we need to prove is that $c_a$ is a bijection and $\lambda: G \to$ Sym$(X)$, given by $a \to c_a$ is a group homomorphism.
First of all let $H=K$. Then we have that $c_a(H) = aHa^{-1} = aKa^{-1} = c_a(K)$ therefore $c_a$ is well-defined. Now let $c_a(H) = c_a(K) \implies aHa^{-1} = aKa^{-1} \implies H=K$, therefore $c_a$ is injection. Now let $H \le G$, as $a^{-1} \in G$ we have that: $c_a(a^{-1}Ha) = (aa^{-1})H(aa^{-1}) = H$. Therefore $c_a$ is surjection and eventually a bijection.
Now for the second part all we need to prove is:
$$\lambda(ab) = \lambda(a)\lambda(b) \iff c_{ab} = c_{a}c_{b}$$
But we have already established this as true in the first part.