Eigenvectors of a symmetric operator on a Hilbert space are orthogonal. But in general they do not form a complete set so that every $L^2(a,b)$ function can be expanded in terms of them. I couldn't find any proof for this statement. Can anyone please suggest me a line for proof or a reference?
Thanks in advance.
In general, it is not true that all the eigenvectors (in the Hilbert space, not "generalized"...) for a self-adjoint (whether bounded or unbounded) operator on a Hilbert space give a Hilbert-space basis for that Hilbert space. It can be that there is no point spectrum at all, but only purely continuous, as with many multiplication operators $Tf(x)=g(x)\cdot f(x)$ where $g$ is essentially bounded: if (for example) these are functions on $\mathbb R$ and $g$ is strictly increasing (but bounded), there are no eigenvalues at all.
But/and your revised question makes me think that this is not what you are asking. That is, on a bounded interval $[a,b]$, many Sturm-Liouville (=good keyword for internet searches) problems (symmetric/self-adjoint second-order differential operators with suitable boundary condiitons) do have pure point spectrum, because the resolvents are compact (self-adjoint) operators. So, yes, there is a Hilbert-space basis consisting of eigenfunctions.
But/and perhaps this is still not quite your question, if you are asking about pointwise convergence of eigenfunction expansions? This is a non-trivial question even in the most classical situation of Fourier series. And many boundary-value problems produce eigenfunctions which are not smooth at the endpoints. E.g. the Dirichlet problem on $[0,2\pi]$ with the Laplacian has eigenfunctions $\sin {n\over 2} x$. For $n$ odd, these do not have differentiable extensions to $2\pi$-periodic functions. Further, $L^2[0,2\pi]$ functions that do not vanish at the endpoints have $L^2$ expansions in terms of these eigenfunctions, but obviously these expansions do not converge at the endpoints, and, therefore, have not-very-rapidly-decreasing coefficients, so the expansions do not converge very well at all.
(I'd claim that many of these seeming disconnects can be understood in terms of Sobolev spaces and Gelfand triples and such, but that's a bit more sophisticated viewpoint than you might want.)