On page 263 of Reed and Simon, book 1, in the formulation of the projection valued measure form of the spectral theorem, they make the following claim which I don't see why is true. That is, the following domain $$D_g = \{ \varphi \ \vert \ \int_{-\infty}^{\infty} \left| g(\lambda)\right|^2 d(\varphi, P_{\lambda}\varphi) < \infty \}$$ is dense.
Note that this is the unbounded, not the bounded case. That is, $g$ is an unbounded function.
On
Identify $H$ with $L^2(\mathbb R)$.
Let $E_{n,k}=\{|g|\leq n\}\cap[-k,k]$. Let $$F_{n,k}=\{\varphi\in L^2(\mathbb R):\ \text{supp}\,\varphi\subset E_{n,k}\}.$$ For each $\varphi\in F_{n,k}$, the integral is finite.
And $\bigcup_{n,k}F_{n,k}$ is dense. Indeed, $$\tag{1}\bigcup_nE_{n,k}=[-k,k].$$ Given $\varepsilon>0$, there exists $k$ and $\varphi_k$, supported on $[-k,k]$, with $\|\varphi-\varphi_k\|<\varepsilon$. By $(1)$, there exists $n$ with $\varphi_k\in F_{n,k}$.
Assume $|g(\lambda)| < \infty$ for all $\lambda$, without assuming that $g$ is bounded. Define an operator $T_g$ by $$ T_gx = \int |g(\lambda)|dP(\lambda)x,\;\;\; x\in D_g, $$ where $D_g$ is as you described in the problem statement. This makes sense because $$ \|T_gx\|^2 = \int |g(\lambda)|^2d(P(\lambda)x,x) < \infty,\;\;\; x\in D_g. $$ The operator $T_g$ is symmetric and non-negative on its domain, and satisfies $$ (T_g+I)\int \frac{1}{1+|g(\lambda)|}dP(\lambda)x = x,\;\; x\in H. $$ To prove $D_g$ is dense in $H$, suppose that $y\in H$ is orthogonal to $D_g$. Then $y=(T_g+I)x$ for some $x\in D_g$, which gives $$ 0 = (y,x)=((T_g+I)x,x) \ge (x,x) \ge 0 \\ \implies x= 0 \\ \implies y=(T_g+I)x=0. $$ Therefore $D_g^{\perp}=\{0\}$, which proves that $D_g$ is dense in $H$.