\begin{equation} \begin{array}{c} \underset{\mathbf{x} \in \mathbb{R}^{m}, \mathbf{y} \in \mathbb{R}^{n}}{\text{minimize}}\hspace{4mm} \big(\lambda_1\left( \mathbf{y}^T V^{(1)}\mathbf{x} \right)^2 + \lambda_2\left( \mathbf{y}^T V^{(2)}\mathbf{x} \right)^2\big) \\ \end{array} \end{equation}
where $\mathbf{V}^{(i)} \in \mathbb{R}^{n\times m}$, and $\lambda_1\geq \lambda_2\geq 0$. How do I prove that the objective function above is not convex?
Actually it's not tricky if we consider the $m=n=1$ case with all constants equal to $1$. The objective expression becomes $2y^2x^2$. The Hessian of this expression is $$\begin{bmatrix} 4y^2 & 8xy \\ 8xy & 4x^2 \end{bmatrix}$$ The determinant of this Hessian is $-48x^2y^2$, which is negative when both $x$ and $y$ are nonzero, so it cannot possibly be positive semidefinite. Hence the objective function is neither convex nor concave.