My professor gave me a list of exercises to complete in preparation for the exam ... but i can't solve this question. I have the feel that the solution is simple but in this moment I can't figure it out.
Question: Let $A \in B(H)$ be self-adjoint. Show that $\lambda \in R(A)$ (where $R(A)$ is the resolvent set of A) if and only if there is an open neighborhood $U \subset \mathbb{R}$ of $\lambda$ with $\chi_{U}(A) = 0$ (whit $\chi$ as indicator function)
I suggest you take a closer look at functional calculi. I will try to present a quick overview.
$\textbf{Continuous functional calculus}$
There exists an isometric $*-$isomorphism (called the continuous functional calculus) $$ f \mapsto f(A) $$ from $\sigma(A)$ onto a closed $C^*$ sub-algebra of $B(H)$, namely, the (commutative) $C^*$ algebra generated by $A$. This map has the property that if $f(z)=p(z,\bar z)$ is a polynomial of (perhaps) two variables, then $f(A) = p(A,A^*)$ is again a "polynomial" in $B(H)$.
$\textbf{Borel functional Calculus}$
One would like to give meaning to $f(A)$ when $f$ is not a continuous function, but instead any bounded Borel function, like the characteristic function of a set. The two definitions should coincide when $f$ is continuous.
For any fixed $x,y \in H$ the function $f \mapsto (f(A)x,y) $ is a bounded functional on $C(σ(A))$ with norm less or equal to $||x|| ~||y||$. By the Riesz representation theorem, there exists a unique finite regular complex Borel measure $μ_{x,y}$ on $\sigma(A)$ with (total variation) norm $||μ_{x,y}|| \le ||x|| ~ ||y||$ such that $$ (f(A)x,y) = \int_{σ(Α)} f~ d \mu_{x,y}, \ \ \ \forall f \in C(σ(Α)). $$ Moreover, if $f$ is a bounded Borel measurable function on $σ(Τ)$, then
$$ H \times H \ni (x,y) \mapsto \int_{σ(Α)} f ~d \mu_{x,y} $$ is a bounded sesquilinear form on $H$ with norm less or equal to $\|{f}\|_{\infty}$. By the Riesz representation theorem, there exists an linear bounded operator, which is again denoted by $f(A)$, such that $$ (f(A)x,y) = \int_{\sigma(A)} f ~ d \mu_{x,y} , \ \ \ \forall x,y \in H .$$ The map $$f \mapsto f(A)$$ from the space of bounded borel functions to $B(H)$ is called the Borel functional calculus, and it extends the continuous functional calculus.
$\textbf{Spectral measures and the Spectral theorem}$
A spectral measure is a map $E$ from the Borel $σ$-algebra of $\sigma(A)$ to the set of projections in $B(H)$ such that
$E(\varnothing)=0$ and $E(\sigma(A))=Id_H$.
$E(S_1 \cap S_2) =E(S_1)E(S_2)$ for all Borel sets $S_1$ and $S_2$,
For each $x,y \in H$, the map $$ S \mapsto E_{x,y} (S) = (E(S)x,y)$$ is a finite regular complex Borel measure on $\sigma(A)$.
Given a spectral measure $E$ and a Borel bounded function $f$ we can define the operator $\int_{\sigma (A)} f d E$ as the unique operator given by the Riesz representation on the sesquilinear form $$ H\times H \ni (x,y) \mapsto \int_K f ~ dE_{x,y} \in \mathbb C .$$ Hence, $\int_K f ~ d E$ is the unique operator satisfying $$ \Big ( \big ( \int_{σ(Α)} f ~ d E\big) x,y\Big )= \int_K f ~ d E_{x,y}. $$
For $S\subset \sigma(A)$ Borel, we let $E(S) = \mathcal X_S (A)$. Then $E$ is a spectral measure with $E_{x,y} = \mu_{x,y}$, the latter being the unique regular complex measures given by The Riesz representation theorem on $C(\sigma(A))$. The Spectral theorem then states that $$A= \int_{\sigma(A)} z d E,$$ where $z$ is the identity function on $\sigma(A)$. That is, $$(Ax,y) = \int_{\sigma(A)} z d E_{x,y}.$$
Back to your problem. If $λ \in U \subset R(A)= \mathbb C \setminus \sigma(A)$, then since for any $x,y \in H$, $$ (\mathcal X_U (A)x,y) = \int_{\sigma(A)} \mathcal X_U dE_{x,y}=0,$$ we get that $\mathcal X_u(A) =0$. For the converse, you need the fact that if $U$ is a relatively open subset of $\sigma(A)$ then $E(U) \equiv \mathcal X_U(A) \neq 0$.