I would really appreciate some examples of algebraic group actions on $\mathbb{P}^n$ with non trivial generic stabilizers and no dense orbit.
The action of the subgroup of the Heisenberg group $H_n$ (see https://en.wikipedia.org/wiki/Heisenberg_group ) consisting of the elements satisfying $b=0$ has these properties, but the orbits are lines. I was hoping I could get an example with more "complicated" orbits.
Edit: the word " generic " has been added.
You might be interested in the flag variety $G/B$ associated with a connected algebraic group with maximal connected solvable subgroup $B$. It's a fact that $G/B$ is a projective variety and may be realized as an orbit in projective space with the properties you listed.
To prove this, one may consider a representation $G \to GL(V)$ such that $B$ is the stabilizer of a line $L \subseteq V$. Then $B$ will be the stabilizer of the point $L$ in the projective space $\mathbb{P}(V)$. By maximality of $B$ in $G$, the orbit of $L$ is closed and so $G/B$ is projective.
In other words, $G/B$ is an example of an orbit of an action by $G$ in projective space that is not dense (since it's closed and if we assume $\operatorname{dim}G$ is large enough), and has the nontrivial stabilizer $B$.
Note also that the Heisenberg group is the group of unipotent matrices $U_3 \subseteq B \subseteq GL_3$ contained in the standard Borel subgroup in $GL_3$ so that $U_3$ is contained in the stabilizer of the action given above on $G/B$.