I'm trying to prove that the projective variety associated to the GIT quotient $$\mathrm{End}(W)^{ss} / H$$ is smooth, where $H$ is a (reductive) subgroup of $\mathrm{Gl}(W)$ acting by pre-composition on $\mathrm{End}(W)$, and $ss$ denotes the semistable points with respect to this action.
I could not find some explicit examples where smoothess of quotients is verified (examples that would help in this situation). Any suggestion for how to handle this problem, and more generally, what are some "standard" techniques / results for proving smoothness of GIT quotients like this one? Also references to some explicit computations would be super appreciated.
Let $S:=Spec(A)$ with $G \subseteq Aut_{k-alg}(A)$ a finite subgroup of the group of $k$-algebra automorphisms of $A$, with $A^G$ the invariant ring. It follows the ring extension $A^G \subseteq A$ is an integral extension and you get a canonical map
$$\pi: S \rightarrow S/G:=Spec(A^G).$$
If $A^G$ is generated as a $k$-algebra by a finite set of elements $g_1,..,g_l \subseteq A$ you get a canonical surjective map of $k$-algebras
$$\phi: k[y_1,..,y_l] \rightarrow A^G$$
defined by $\phi(y_i):=g_i$, and an isomorphism $A^G \cong k[y_i]/I$ where $I:=ker(\phi)$ is the kernel of $\phi$. Hence $S/G \subseteq \mathbb{A}^l_k$ is an affine algebraic variety which may be singular. There are explicit examples in the litterature where such quotients are constructed (singular and non-singular quotients) and you find them at mathscinet or the zentralblatt search site. Similar constructions exist for positive dimensional groups. The "smoothness" of the affine variety $S/G$ depends on the ideal $I$ of relations between the generators $g_i$, and I do not believe there is a general criteria for this. If the ring $A$ has an $\mathbb{N}$-grading and the action of $G$ has the property that the ideal $I$ is a homogeneous ideal, the quotient $S/G$ may have singularities.
Question: "I could not find some explicit examples where smoothess of quotients is verified."
Answer: If $G:=SL(V)$ where $V$ is an $n$-dimensional complex vector space and $W \subseteq V$ is a $d$-dimensional subspace with $P \subseteq G$ the subgroup fixing $W$ it follows the quotient $G/P \cong \mathbb{G}(d,V)$ is the grassmannian variety of $d$-planes in $V$ and $G/P$ is a smooth algebraic variety. More generally if $W$ is a finite dimensional vector space over a field $k$ and your action satisfies $End(W)^{ss}=G$ for some closed sub-group $G \subseteq GL(W)$, it follows the quotient $G/H$ is a smooth quasi projective variety of finite type over $k$. You find a similar question here:
Quotient of a Lie algebra by a subalgebra - what is it?
If you look at Borel's book "Linear algebraic groups" you find in Lemma 1.8 (the closed orbit lemma) the full argument. Essentially if an algebraic group $G$ acts on a variety $V$ then each orbit $Gx$ is smooth. For simplicity: Given any pair of points $y,z \in Gx$ there is an element $g\in G$ with $gz=y$ since $y,z$ is in the same orbit. Since $g: Gx \rightarrow Gx$ is an automorphism of $Gx$ it follows the local rings $\mathcal{O}_{Gx,y}\cong \mathcal{O}_{Gx,z}$ are isomorphic. Hence any two points $y,z$ in the orbit have isomorphic local rings. Generically $Gx$ is smooth and hence it follows $Gx$ is smooth at any point $y$. If a quotient $G/H$ exists with a left $G$-action, it follows any two points $x,y\in G/H$ will have isomorphic local rings
$$\mathcal{O}_{G/H,x}\cong \mathcal{O}_{G/H,y}$$
hence $G/H$ is smooth at any point. The existence of the quotient $G/H$ is proved in Theorem 6.8 in the Borel book. If $H\times F \rightarrow F$ is a left $H$-action where $F$ is smooth, you may construct a locally trivial fibration with fiber $F$
$$ \pi:G\times F/H \rightarrow G/H,$$
and $G\times F/H$ is smooth.
In general if $\sigma: G\times X \rightarrow X$ is an action with one orbit $Gx$ it follows $X$ is smooth by the same argument.
Note: The "stacks project" is constructing "quotients" in a more general setting, you may be aware of this site.