Consider $$\begin{aligned} \dot{x} &= ax + by\\ \dot{y} &= bx + ay\end{aligned}$$ where $a>0$ and $b<0$. Find the fixed points and classify them.
I haven't been able to find any fixed points other than the trivial $(x,y)=(0,0)$ solution. Any help would be appreciated!
Fixed points are going to be whatever coordinates in the plane that make the derivative zero. If you set this up as a matrix differential equation you get $$ \frac{d}{dt} \left [ \begin{array}{c} x\\ y\\ \end{array} \right ] \;\; =\;\; \left [ \begin{array}{cc} a & b \\ b & a \\ \end{array} \right ]\left [ \begin{array}{c} x\\ y\\ \end{array} \right ] $$
and you can diagonalize this matrix as
$$ \left [ \begin{array}{cc} a & b \\ b & a \\ \end{array} \right ] \;\; =\;\; \underbrace{\left [ \begin{array}{cc} 1/\sqrt{2} & -1/\sqrt{2} \\ 1/\sqrt{2} & 1/\sqrt{2} \\ \end{array} \right ]}_{ = P} \underbrace{\left [ \begin{array}{cc} a + b & 0 \\ 0 & a-b \\ \end{array} \right ]}_{=D} \underbrace{\left [ \begin{array}{cc} 1/\sqrt{2} & 1/\sqrt{2} \\ -1/\sqrt{2} & 1/\sqrt{2} \\ \end{array} \right ]}_{=P^T}. $$
Notice that since $a>0$ and $b<0$ then the only way for this matrix to be singular is if $b = -a$. This would make the eigenvalue $a+b = 0$, and the corresponding eigenvector to this is $\left [ \begin{array}{cc} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \end{array} \right ]^T$. Therefore we get two cases for fixed points:
$$ \textbf{Fixed Points} \;\; =\;\; \begin{cases} \text{Span} \left \{\left [ \begin{array}{c} 1 \\ 1 \\ \end{array} \right ] \right \}, & \text{if} \; b = -a \\ \left [ \begin{array}{c} 0 \\ 0 \\ \end{array} \right ], & \text{otherwise} \\ \end{cases}. $$