If we model a marble moving on a saddle
$$
V(x,y)= \frac{1}{2}m\omega^2 (x^2-y^2)
$$
and set this saddle in rotation about $\hat z$ with angular velocity $\Omega$, one obtains (in the regime where $\Omega/\omega \ll 1$) the coupled equations
of motion
\begin{align}
m\omega^2\ddot{x}&= m\Omega \dot{y}-m\omega^2 x\, \\
m\omega^2\ddot{y}&= -m\Omega \dot{x}+m\omega^2 y\, \, . \tag{1}
\end{align}
where we use dimensionless time $\tau=\omega t$ and $\dot{x}=\frac{dx}{d\tau}$ etc.
I want to analyze the stability of this motion near the origin. Clearly for $\Omega=0$ it is unstable as the inverted potential in the $y$ direction is an unstable fixed point.
Now in
David, Guery-Odelin, and Lahaye Thierry. Classical Mechanics Illustrated By Modern Physics: 42 Problems With Solutions. World Scientific Publishing Company, 2010
they suggest looking for a solution of the form \begin{align} x=x_0 e^{\lambda \tau}\, ,\qquad y_0=y_0 e^{\lambda \tau}\, \tag{2} \end{align} and check if there are solutions with $\lambda$ purely imaginary, which would convert using Euler's theorem to a bounded solution in terms of $\sin$ and $\cos$. In the regime of (1) there is no bounded solution: one inserts the ansatz into the equation of motion to obtain a set of coupled equations now of the form \begin{align} \left(\begin{array}{cc} \lambda^2\omega^2+\omega^2&-\lambda\omega \Omega \\ \lambda \omega\Omega & \lambda^2\omega^2-\omega^2 \end{array}\right)\left(\begin{array}{c} x_0 \\ y_0\end{array}\right) :=M\left(\begin{array}{c} x_0 \\ y_0\end{array}\right) = 0\, , \end{align} The result follows by solving for $\lambda^2$ from the condition $\hbox{Det}(M)=0$, where one root is positive so that $\lambda$ is real.
It seems the ansatz of (1) is quite restrictive as the resulting frequency of motion in both $x$ and $y$ would be the same.
How does one analyze the stability of (1) in general? Can one use the more general ansatz \begin{align} x=x_0 e^{\lambda_1 \tau}\, ,\qquad y_0=y_0 e^{\lambda_2 \tau}\, \tag{2} \end{align} and find simultaneous purely imaginary solutions to a determinental-like equation in $\lambda_1,\lambda_2$?