$$ \max_{\pmb x} \cfrac{\pmb x^H \pmb P \pmb P \pmb x} {\pmb x^H \pmb P \pmb x + \sigma^2}$$ s.t. $$ ||\pmb x ||^2_2 < N$$ where $\pmb x$ is a vector,$\pmb P$ is symmetric positive definite matrix, $\sigma^2$ and $N$ are both constants.
Any comments would help, thanks!
Here is how to solve the problem with the constraint $\|\mathbf x\|^2\le N$ instead of $\| \mathbf x\|^2<N.$
Let the $i$-th eigenvector of $P$ be $\mathbf v_i$, with eigenvalue $p_i$; assume $\|\mathbf v_i\|=1$ for all $i$. Your solution will be of the form $\mathbf x=\sum a_i \mathbf v_i$ where $\sum a_i^2 < N$, and your objective function will be $\sum a_i^2 p_i^2/(\sum a_i^2 p_i+\sigma^2)$. So let $j$ maximize $p_j^2/(p_j+\sigma^2)$ and let $a_i=0$ for all $i\ne j$ and let $a_j=\sqrt N$.
To see why, contemplate the following picture. Let $S=\{(p_i+\sigma^2,p_i^2)\}\cup \{(\sigma^2,0)\}\subset \mathbb R^2$, and let $C$ be the convex hull of $S$. Your optimum is given by the point $(x,y) \in C$ maximizing the ratio $y/x$; it is easy to see that this occurs at a point in $S$. The convex set $C$ is the set of all possible $(\sum_i a_i^2 p_i^2, \sum_i a_i^2 p_i+\sigma^2)$ values; the business with the point $(0,\sigma^2)$ takes care of the possibility that $\sum a_i^2<N$.
In general the maximum is not attained in the original, $\|\mathbf x\|^2<N$ problem.