I want to define a proper domain $D(H) \subset L^2$ for this Hamiltonian ( $\theta$, $\phi$ are the standard angles in spherical coordinates). Furthermore, the wave function is supposed to satisfy that the function itself and its derivative is continuous at the boundary of the intervals of $\theta$ and $\phi$.)
$$ \left\{ -\frac{1}{\sin^2\theta} \left[
\sin\theta\frac{\partial}{\partial\theta} \Big(\sin\theta\frac{\partial}{\partial\theta}\Big)
+\frac{\partial^2}{\partial \phi^2}\right]\right\} \psi(\theta,\phi)+V(\theta)\psi(\theta,\phi) = E \psi(\theta,\phi),$$
where $V \in C^{\infty}([0,2\pi]) \rightarrow \mathbb{R}$.
One normally wants that this operator shall be self-adjoint (therefore it should be a dense domain). Could anybody explain to me how to find a proper domain for this operator?
Hilbert Space: You have not specified the underlying Hilbert space $X$, but I assume you want $X=L^{2}_{\sin\theta}([0,\pi]\times[0,2\pi])$ with weighted inner-product $$ (f,g) = \int_{0}^{2\pi}\int_{0}^{\pi}f(\theta,\phi)\overline{g(\theta,\phi)}\sin\theta\,d\theta\,d\phi. $$ The operator $$ Lf = -\frac{1}{\sin\theta}\frac{\partial}{\partial\theta}\sin\theta\frac{\partial}{\partial\theta}f-\frac{1}{\sin^{2}\theta}\frac{\partial^{2}}{\partial\phi^{2}}f $$ is of primary concern. I expect a selfadjoint domain for $L$ to also work for $L+V$ because $V$ will be relatively bounded for any reasonable domain. $L$ can be viewed as a restriction of the Laplacian.
Domain: Let $L_{0}$ be the formal operator $L$ restricted to $C^{\infty}$ functions on $[-1,1]\times[0,2\pi]$ which are periodic (along with all derivatives) in $\phi$, and which satisfy $$ f,\frac{\partial f}{\partial \theta}, \frac{1}{\sin\theta}\frac{\partial f}{\partial\phi},Lf \in X. $$ This is a fairly natural domain because $$ \begin{align} \sin\theta(Lf)\overline{f} & = -\left(\frac{\partial}{\partial\theta}\sin\theta\frac{\partial f}{\partial\theta}\right)\overline{f}-\frac{1}{\sin\theta}\frac{\partial^{2}f}{\partial\phi^{2}}\overline{f} \\ & = -\frac{\partial}{\partial\theta}\left(\sin\theta\frac{\partial f}{\partial\theta}\overline{f}\right)+\sin\theta\left|\frac{\partial f}{\partial\theta}\right|^{2} -\frac{\partial}{\partial\phi}\left(\frac{1}{\sin\theta}\frac{\partial f}{\partial\phi}\overline{f}\right)+\frac{1}{\sin\theta}\left|\frac{\partial f}{\partial\phi}\right|^{2}, \end{align} $$ which leads to $$ (L_{0}f,f) =\lim_{\epsilon\downarrow 0}\int_{0}^{2\pi}\int_{\epsilon}^{\pi-\epsilon}(Lf)\overline{f}\sin\theta\,d\theta\,d\phi = \left\|\frac{\partial f}{\partial\theta}\right\|^{2} +\left\|\frac{1}{\sin\theta}\frac{\partial f}{\partial\phi}\right\|^{2},\;\; f \in \mathcal{D}(L_{0}). $$ It also follows that $(L_{0}f,g)=(f,L_{0}g)$ for all $f,g\in\mathcal{D}(L_{0})$.
There are reasons why this domain is the correct one for Quantum; for one thing, the operators $\frac{\partial}{\partial\theta}$ and $\frac{1}{\sin\theta}\frac{\partial}{\partial\phi}$ are defined, and these are useful operators related to momentum; these operators are needed to give a sum-of-squares factoring for the Hamiltonian. This domain is also suitable for the spherical harmonics. This domain is related to the Friedrich's extension, which is a compelling reason to consider it from the viewpoint of Mathematics and of Energy. Another important fact is that this domain eliminates the function $$ \delta_{0}(\theta)=\ln\left[\frac{1+\cos\theta}{1-\cos\theta}\right], $$ which satisfies $L\delta_{0}=0$, even though $\delta_{0}\in X$, and $L\delta_{0}\in X$. No Quantum selfadjoint operator should have $\delta_{0}$ in the domain. There are boundedness conditions you can impose that will imply the above, but the $L^{2}$ conditions I stated can be imposed on the Sobolev space in a natural way, and these conditions give a complete description of a domain on which $L$ is selfadjoint in the strictest sense.