Consider a linear system $\dot{x} = Ax + Bu$ with a quadratic performance index: \begin{equation} J = \frac{1}{2}\int_{t_0}^{\infty}[x(t)^TQx(t) + u(t)^TRu(t)]dt \end{equation}
Now suppose any one of the states is unstable. Under this assumption, does this performance index make sense? Will J become infinite? All the solutions I see are working under the assumption that $x(t)$ is stable. But why can't we choose an $u$ to compensate unstable effect of $x(t)$?
Your cited source states that the system (3.4.1)
$$ \dot{x}=A\,x+B\,u $$
has to be completely controllable.
That would be a sufficient condition to ensure that an optimal solution has a finite value for the performance index, but it is not a necessary condition.
A necessary condition in order for the performance index to be finite using an the optimal solution and that the closed loop system is stable, is that $(A,B)$ is stabilizable and $(A,Q)$ is detectable, for example see [1]. The first requirement about stabilizability implies that all unstable modes of $A$ can be stabilized by $u(t)$ through $B$ and the second requirement about detectability implies that all unstable modes of $A$ contribute to a non-zero amount to the performance index through $Q$.
[1] Skogestad, Sigurd, and Ian Postlethwaite. Multivariable feedback control: analysis and design. john Wiley & sons, 2005.