Consider the system: $$\left \{\begin{align}\dot x &= x-y-x(x^2+y^2)\\ \dot y &= x+y-y(x^2+y^2) \end{align}\right.$$ Find the fundamental matrix solution $Y(t)$ explicitly, assuming $x(0) = \cos \phi_0, y(0) = \sin \phi_0$.
I know the system can be written as:
$$\left \{\begin{align}\dot r &= r-r^3\\ \dot \phi &= 1 \end{align}\right.$$
Whereas I'm looking for $Y(t)$ such that: $\dot Y(t) = Df(x) \cdot Y(t)$ (and $Y(0) = I$. I guess I should use the system in polar coördinates then:
$$\begin{pmatrix}\dot y_{11}(t) & \dot y_{12}(t) \\ \dot y_{21}(t) & \dot y_{22}(t)\end{pmatrix} = \begin{pmatrix}1-3r(t)^2 & 0\\ 0 & 0\end{pmatrix} \cdot \begin{pmatrix}y_{11}(t) & y_{12}(t) \\ y_{21}(t) & y_{22}(t)\end{pmatrix} $$
Then I should solve $\dot y_{11}(t) = (1-3r(t)^2)\cdot y_{11}(t)$ but how can I do this?
Or should I use the original system, where the Jacobian matrix seems complicated?
After solving $r'=r-r^3$ you will have $r(t)$ and $\theta(t)$.
This will give you $x=r\cos\theta$ and $y=r\sin\theta$ as functions of $t$ and of the initial condition $(x(0),y(0))=(x_0,y_0)$. The best way to proceed is to write down the solutions in the form $$ (x(t),y(t))=\varphi_i(x_0,y_0), $$ since then $Y(t)$ is simply the Jacobian $d_{(\cos\phi_0,\sin\phi_0)}\varphi_t$.
PS: One should note that it is not fine to talk about fundamental matrices of nonlinear systems, although of course there is only one reasonable interpretation of your question.