I have the following system $$ X'=a-(b+1)X+X^2Y,\\ Y'=bX-X^2Y. $$ I have to prove that $(a,b/a)$ is an equilibrium point if $a^2+1>1b$ and unstable if $a^2+1<b$.
Also i have to prove that using $b$ as a parameter and fixing $a$, the point is in the conditions for the Poincare-Andronov-Hopf theorem.
Also i have to analyse the stability at the origin when $a^2+1=b$ (in the normalized system) and to find the bifurcation curve.
This is my work so far: Firstly, we perform the variable change $u = X - a$, $v = \frac{Y - b}{a}$ to bring the point $(a, \frac{b}{a})$ to the origin, obtaining the system:
\begin{align*} u' &= (b-1)u + \frac{b}{a}u^2 + a^2v + u^2v + 2auv, \\ v' &= -bu - u^2v - 2auv - a^2v - \frac{b}{a}u^2. \end{align*}
The Jacobian of the system is given by:
$$ J(a, b) = \begin{pmatrix} b-1 & a^2 \\ -b & -a^2 \end{pmatrix}. $$
The eigenvalues of the system are:
$$ \lambda_{\pm} = \frac{(-a^2 + b - 1) \pm i\sqrt{-(a^4 - 2a^2 + b^2 - 2a^2b - 2b + 1)}}{2}. $$
Since they are complex eigenvalues, we need to check the sign of the real part, i.e., the sign of $-a^2 + b - 1$. In this case, we have: If $a^2 + 1 < b$, the origin is unstable. If $a^2 + 1 > b$, the origin is stable. If $b = a^2 + 1$, we cannot decide.
In this last case, we are in the conditions of a Hopf bifurcation for the parameter $b$. Specifically, we have a system of the form:
$$ \vec{u}' = A(b)\vec{u} + F(b, \vec{u}), $$
where $F(b, (0, 0)) = D_{u,v}F(b, (0, 0)) = 0$, and the eigenvalues of $A(b)$ are of the form $\alpha(b) \pm i\beta(b)$, with $\beta(a^2 + 1) = 4a^2 > 0$, $\alpha(a^2 + 1) = 0$, and $\frac{d\alpha}{db}(a^2 + 1) = \frac{1}{2} \neq 0$. Therefore, the point $(0, 0)$ for $b = a^2 + 1$ satisfies the conditions of the Poincaré-Andronov-Hopf Theorem.
I'm totally lost with the last two questions. Any help will be very appreciated!