The system of ODEs
$$\dot{u} = bu - v + au(u^2 + v^2)$$ $$\dot{v} = u + bv + av(u^2 + v^2)$$
can be written in polar coordinates as
$$\dot{r} = br + ar^3$$ $$\dot{\phi} = 1$$
I know that in Euclidean coordinates I can calculate the Jacobian and evaluate it at the equilibrium (0,0) to get
$$ \begin{pmatrix} b & -1 \\ 1 & b \end{pmatrix}$$
How can I get the correct Jacobian directly from polar coordinates? Does this even make sense? I should at-least be able to get a nonsingular Jacobian at the origin right?
Ordinarily, you can get the correct Jacobian directly by taking the partial derivatives as you had done for $u,v$. This will yield a polar Jacobian matrix of
$$ J=\begin{pmatrix} b & 0 \\ 0 & 0 \end{pmatrix} $$
This leads to your question about invertibility. Ordinarily, if the rank of the Jacobian should remain the same under any change of coordinates, since we are in theory multiplying our Jacobian matrix on the left and right by invertible matrices (these invertible matrices correspond to the Jacobian matrix of the coordinate transformation), but in this case we lose a dimension. So what's going on?
This is one of the rare instances where polar coordinates fails us, namely at the origin. If we were examining the Jacobian at any other point in $(u,v)$ space then there would be no issue. The problem comes from the lack of invertibility of the coordinate at the origin. The transformation
$$ u=r\cos\phi,\;\;\;\;v=r\sin\phi $$
does not uniquely identify $(u,v)=(0,0)$ with a unique $(r,\phi)$, since we may let $\phi$ be anything we want so long as $r=0$, meaning the transformation is not one to one. Thus, the coordinates actually are not useful at the origin. This appears in the malfunction of the Jacobian matrix at the origin and why you are losing invertibility.