I have a small question please, how to prove that this set:
$F=\lbrace h\in H, \langle f''(u)h,h\rangle <0\rbrace$ is a sub space of the Hilbert space $H$, where $f''(u)$ is a self-adjoint operator from $H$ to $ H$
I calculated $\langle f''(u)(\lambda h+\mu k),(\lambda h+\mu k) \rangle= \langle \lambda f''(u) h,\lambda h\rangle +\langle \lambda f''(u) h,\mu k\rangle+ \langle \mu f''(u) k,\lambda h\rangle + \langle \mu f''(u) k,\mu k\rangle$ But i don't know how to say that $\langle \lambda f''(u) h,\mu k\rangle <0$ and $\langle\mu f''(u) k,\lambda h \rangle <0 $
please thank you
It is not a subspace. If your operator is positive definite, then $F$ is empty. More generally, your $F$ cannot contain $h=0$, so it will not be a subspace for any choice of an operator, selfadjoint or not.
Even if you replace $<$ with $\leq$, it will still not be a subspace in general: consider the operator $T=\begin{bmatrix} 0&1\\1&0\end{bmatrix}$. For $h=(x,y) $ you have $$ \langle Th,h\rangle=\left\langle\begin{bmatrix} 0&1\\1&0\end{bmatrix}\begin{bmatrix} x\\y\end{bmatrix},\begin{bmatrix} x\\y\end{bmatrix}\right\rangle=2xy. $$ So $(2,-1)$ and $(-1,2)$ are in $F$, while $(2,-1)+(-1,2)=(1,1)$, not in $F$.