I want to solve the following problem, \begin{equation} \begin{aligned} & \underset{\mathbf{x}}{\text{minimize}} & & \mathbf{x^T}\mathbf{Px} \\ & \text{subject to} & & \mathbf{A{x}}=\mathbf{0} \\ &&& \mathbf{{x}^{T}}\mathbf{{x}}=1\\ &&& \mathbf{{x}^{T}}\mathbf{S}\mathbf{{x}}= n_0\\ &&& \mathbf{{x}^{T}}\mathbf{T}\mathbf{{x}} \le k_0\\\end{aligned} \end{equation} Where $x\in \mathbb{R}^n, n_0,k_0 \in \mathbb{R^+}, P \in {S_{++}}(n) $
Is it possible to solve this problem ? If yes, then how?
What about the matrices $T$ and $S$? Are they positive semi-definite?
In any case this problem is not convex due to the constraints $x^T x =1$ and $x^TSx=n_0$. So it is in general a difficult problem.
There might be some specialized algorithm since you are optimizing on the unitary sphere. In general, any nonlinear solver should give you at least a stationary point, if any (it might be infeasible). Otherwise you need to look to deterministic Branch-and-Bound (see Couenne for instance).