can you help me to solve this exercice? The first point is ok but I have problems with the others. Let $S_\varepsilon$ an continuous operator on $L^2(\mathbb{R})$ define as $$ S_\varepsilon[f](x)= \sin(\frac{x}{\varepsilon}) f(x) $$
i) show that is self-adjoint; [Solution. Simple: $(S^*_\varepsilon[g],f)=(g,S_\varepsilon[f])=(S_\varepsilon[g],f)$, and the symmetry of $(,)_{L^2(\mathbb{R})}$]
ii) Find the spectrum (point, residual, compression)
iii) show that $||S_{\varepsilon}[f](x)||\nrightarrow 0$
iv) show that $||S_{\varepsilon}[f](x)||\rightharpoonup0$
A selfadjoint operator has no residual spectrum. Let us write $h=\sin x/\varepsilon$, as the argument is the same for any choice of function. If $\lambda$ were an eigenvalue, you would have $\lambda f = hf$, or $$ (\lambda - h) f=0 $$ (all equalities are "a.e."). So you need $f=0$ wherever $\lambda-h\ne0$. As the sets of zeroes of $h-\lambda$ has measure zero, you get that $f=0$ a.e. So $S$ has no eigenvalues, and so the whole spectrum belongs to the pure approximate point part.
Using the same idea it is easy to see that the spectrum of $S$ is the (essential) range of $h$. In this case $h$ is continuous so it is the actual range: $$\sigma(S_\varepsilon)=[-1,1].$$
For the norm convergence, take $$f=\tfrac{n}{\sqrt2}\,1_{I_n},\ \ \ \text{ where } \ I_n=\left(\tfrac1n\left(\tfrac\pi2-\tfrac1n\right),\tfrac1n\left(\tfrac\pi2+\tfrac1n\right)\right). $$ Then $\|f\|=1$ and $$ \|S_{1/n}f\|^2=\frac{n}{\sqrt2}\int_{I_n}\sin^2{nx}\,dx\geq\sin\left(\tfrac\pi2-\tfrac1n\right). $$ Thus $\|S_{1/n}\|\to1$, making it impossible for $\|S_\varepsilon\|\to0$.
Item $iv$ as written makes no sense. I assume it means $S_\varepsilon f\to0$ weakly. Assume first that $f=1_{[-k,k]}$, $g=1_{[-m,m]}$, with $c=\min\{m,k\}$. Then
$$ |\langle S_\varepsilon f,g\rangle|=\left|\int_{-c}^c \sin\tfrac x\varepsilon\, dx\right| =\varepsilon\left|\int_{-c/\varepsilon}^{c\varepsilon}\sin v\,dv \right| =2\varepsilon|\cos \varepsilon c|\leq2\varepsilon\to0. $$ Since steps functions are linear combination of $f$ or $g$ as above, we get $\langle S_\varepsilon f,g\rangle\to0$ for all $f,g$ step functons. Now for arbitrary $f,g$ we may approximate by step functions (the step functions are dense, since they approximate the continuous functions with compact support, and these are dense in $L^2(\mathbb R)$) and still show that $\langle S_\varepsilon f,g\rangle\to0$.