The problem is: find
$$\min \mathrm{P}\left[{\log(1+p||H^H \mathbf{w}||^2)\over 1+p||G^H \mathbf{w}||^2}<R\right]$$
constraint to: $||\mathbf{w}||^2=1$
where $H$ and $G$ are matrices of dimension $n\times n$ while $\mathbf{w}$ is a vector of dimension $n\times 1$. $p$ is a positive scalar. $R$ is a threshold value which is positive. $\mathrm{P}$ refers to probability. The variable is $\mathbf{w}$. How can we decide whether this problem is convex or not?
Convexity is the exception, rather than the rule. If everything isn't just so in a mathematical program, it is very likely non-convex. In practice, if you did not build your model with convexity in mind from the outset, the likelihood that you will arrive at a convex result is nearly zero.
For this specific example, the nonlinear equality constraint $\|w\|^2 = 1$ is an immediate deal-breaker. This describes a non-convex constraint set.
Your objective function is almost certainly not convex as well. There may be no way to know for sure, because you haven't specified a probability distribution. But the function on the left-hand side of the inequality is not convex, and I am reasonably sure that the inequality itself does not describe a convex set.