A vector field $X:\mathbb{R}^3 \rightarrow \mathbb{R}^3$ is defined by the following formula:
$$ X \begin{pmatrix} x\\ y\\ z\\ \end{pmatrix} = \begin{pmatrix} -y(x^2+y^2)\\ x(x^2+y^2)\\ (1-x^2-y^2) \end{pmatrix}$$
Determine formulas for the solution curves $y:\mathbb{R}\rightarrow \mathbb{R}^3$ of the differential equation $ \dot{y}(t)=X(y(t))$ with the initial condition $$y(0) = \begin{pmatrix} 0\\ 0\\ 0\\ \end{pmatrix}$$
Unfortunately, I don't really know how to start. Do I have to find out the eigenvalues? Or do I have to calculate the derivates? How I get the characteristic polynomial?
So we essentially want to solve $$ \begin{pmatrix} \dot x(t)\\ \dot y(t)\\ \dot z(t)\\ \end{pmatrix} = \begin{pmatrix} -y(t)(x(t)^2+y(t)^2)\\ x(t)(x(t)^2+y(t)^2)\\ 1-x(t)^2-y(t)^2 \end{pmatrix}$$ Let $x(t) = r(t)\cos\theta(t)$ and $y(t) = r(t) \sin\theta(t)$. I will omit the argument $t$ from now on and just simply write $r$ for $r(t)$ etc.
From $r^2 = x^2 +y^2$, we see that $\dot r r = \dot x x + \dot y y,$ and inserting $\dot x$ and $\dot y$ from our system yields \begin{align} \dot r r = -xy(x^2+y^2) + yx(x^2+y^2) = 0 \quad \Rightarrow \quad \dot r = 0. \end{align}
From $\theta = \arctan y/x$, we see that $$\dot\theta = \frac{\dot y x - \dot x y }{x^2+y^2} = \frac{\dot y x - \dot x y }{r^2},$$ and hence $$ \dot\theta = \frac{1}{r^2}(x^2(x^2+y^2)+y^2(x^2+y^2)) = x^2+y^2 = r^2. $$
Hence, in the coordinates $(r,\theta, z)$, our system becomes $$ \begin{pmatrix} \dot r\\ \dot \theta\\ \dot z\\ \end{pmatrix} = \begin{pmatrix} 0\\ r^2\\ 1-r^2 \end{pmatrix}.$$
The first equation implies that $$ r(t) = A, \quad A \in \mathbb{R}. $$ Then the second equation reads $\dot\theta = A^2,$ which yields $$ \theta(t) = A^2t+ B, \quad B\in\mathbb{R}.$$ The third one becomes $\dot z = 1-A^2,$ so $$ z(t) = (1-A^2)t+C \quad C\in\mathbb{R}. $$
Transforming back to our original coordinates, we see that the general solution of the system is $$ \begin{pmatrix} x(t)\\ y(t)\\ z(t)\\ \end{pmatrix} = \begin{pmatrix} A\cos(A^2t+B)\\ A\sin(A^2t+B)\\ (1-A^2)t+C \end{pmatrix}. $$ You should be able to find the values of the constants by using the initial condition.
EDIT: I might have made a mistake somewhere, but the initial condition seems to imply that the solution is $(x,y,z) = (0,0,t)$.
EDIT 2: Actually now that I look at the differential equation again, it makes sense that this is the solution given the initial condition.
In fact, solving the equation explicitly was not even necessary. We could have just noted that, at time $t=0$, $\;\dot x = \dot y = 0$, and $\dot z = 1$, so $x$ and $y$ stay in place, while $z$ increases at a constant speed.
Furthermore, since $\dot x$ and $\dot y$ are independent of $z$, they will stay both stay in $0$ at all times, and since $\dot z$ is independent of $x$ and $y$, $\;z$ will keep increasing at unit speed.