Given the operator $T (\psi)(x):= \psi''(x)-2q \cos(2x)\psi(x)$ with $T : D(T) \subset L^2[0,2\pi] $
I was wondering: What is the right domain $D(T)$ for this operator if we want to solve the Mathieu equation Mathieu equation given by $$T(\psi) = \lambda_n \psi,$$ where $\lambda_n$ is the n-th eigenvalue of the Sturm-Liouville problem with boundary conditions:
$$\psi(0)= \psi(2 \pi),\psi'(0)= \psi'(2 \pi).$$
For sure, we want to take the domain in such a way, that the operator is self-adjoint and all possible solutions are in this domain. Therefore, I was wondering whether there is a canonical domain for this problem?
By the way: I would also be interested in the argument. Why this operator is self-adjoint on this domain. Probably, there is one standard argument why this is the case for all periodic sturm-liouville problems.
The operator $T\psi = \psi''-2q\cos(2x)\psi$ is a regular Sturm-Liouville operator because the coefficient of the highest order term does not vanish on the interval, and the other coefficients are nice on the interval. The domain $\mathcal{D}(T)$ is almost suggested in the comments. The Sobolev space consists of all twice-absolutely continuous functions on $[0,2\pi]$, and you need to add periodic conditions: $$ \mathcal{D}(T)=\{ f \in L^{2} : f, f' \in \mathcal{AC}[0,2\pi],\; f(0)=f(2\pi),\; f'(0)=f'(2\pi),\;\; f'' \in L^{2}. \}. $$ For $f,g \in \mathcal{D}(T)$ it is easy to verify that $(Tf,g)=(f,Tg)$ where $(\cdot,\cdot)$ is the usual inner-product on $L^{2}[0,2\pi]$. It's not quite as obvious that $T$ is selfadjoint, and not just symmetric. However, this can be proved by showing that $(T\pm iI)$ are surjective. Equivalently, assume $g \in L^{2}[0,2\pi]$, and you must show that $$ Tf_{1}-if_{1}=g,\;\; Tf_{2}+if_{2}=g $$ have solutions $f_{1},f_{2}\in \mathcal{D}(T)$. Such a result can be established in a classical way using variation of parameters on eigenfunctions with eigenvalues $\pm i$.
To carry out this plan using classical ODE theory, suppose $\phi_{i}$, $\psi_{i}$ are linearly independent classical solutions of $$ -w''+2q\cos(2x)w-iw=0. $$ Then the Wronskian $W(\phi_{i},\psi_{i})=\phi_{i}\psi_{i}'-\psi_{i}\phi_{i}'$ is constant and non-zero. Rescale the solutions, if necessary, to obtain $W\equiv 1$ on $[0,2\pi]$. Then $$ f=\phi_{i}(x)\int_{0}^{x} \psi_{i}(u)g(u)\,du+\psi_{i}(x)\int_{x}^{2\pi}\phi_{i}(u)g(u)\,du $$ is a solution of $-f''+2q\cos(2x)f-if = g$ because $$ \begin{align} f' & =\phi_{i}'\int_{0}^{x}\psi g\,du+\psi_{i}'\int_{x}^{2\pi}\phi_{i}g\,du,\\ f'' & =\phi_{i}''\int_{0}^{x}\psi g\,du+\psi_{i}''\int_{x}^{2\pi}\phi_{i}g\,du + \phi_{i}'\psi g-\psi_{i}'\phi g \\ & =\phi_{i}''\int_{0}^{x}\psi g\,du+\psi_{i}''\int_{x}^{2\pi}\phi_{i}g\,du-g \end{align} $$ The general solution of $-f''+2q\cos(2x)f-if=g$ is then $$ f = A\phi_{i}+B\psi_{i}+\phi_{i}(x)\int_{0}^{x} \psi_{i}(u)g(u)\,du+\psi_{i}(x)\int_{x}^{2\pi}\phi_{i}(u)g(u)\,du. $$ What remains is to show that there exist constants $A$ and $B$ such that $f(0)=f(2\pi)$ and $f'(0)=f'(2\pi)$. For such $A$, $B$, one has $f \in \mathcal{D}(T)$ and $(T-iI)f=g$. It then follows that $T=T^{\star}$ on the stated domain $\mathcal{D}(T)$.