Consider the system
$$\dot{\vec{x}} = - \nabla U(\vec{x})$$
where $U(x,y)=x^2+y^2 + a xy$, for $a \in \mathbb{R}$.
For which values of $a$ are the solutions $t \mapsto (x(t),y(t))$ periodic?
**My attempt: **
I recast it as $\vec{x}' = A\vec{x}$, where $A=\begin{bmatrix} -2 & -a \\ -a & -2 \end{bmatrix}$.
I know that the behaviour can be determined by diagonalization, i.e. $$ \vec{x}(t)=Q \operatorname{diag} \Bigl( e^{\lambda_1}, e^{\lambda_2)} \Bigr) Q^{-1}\vec{x_0}$$
I determine $\lambda_{1,2}=-2 \pm |a|$
Therefore the solutions $\Bigl( x(t),y(t)\Bigr) = \Bigl( e^{(-2+|a|)t}, e^{(-2-|a|)t} \Bigr) \vec{x_0}$
In particular, no matter what values of $a \in \mathbb{R}$ I have: the solutions won't be periodic.
Yes, that is normal. The negative gradient field points down in the height field of $U$. Formally, $$ \frac{d}{dt}U(x(t))=U'(x(t))\dot x(t)=-\|∇U(x)\|^2. $$
In a periodic solution the value of $U$ would have to return to the starting value after one period. That is only possible if $\|∇U(x)\|=0$, thus $\dot x=0$, which gives a constant solution.