Given the potential $U_{pot} = \frac 1 2 k r^2$ where $\vec r = (x,y)$, $r=\sqrt{x^2+y^2}$ we can calculate the force
$$\vec F = - \nabla U_{pot} = -2k \frac{1}{2} \frac{1}{\sqrt{x^2+y^2}}2(x,y) = -2k \frac 1 r \vec r \qquad\qquad (1)$$
Is this correct?
Now with Newton, we can construct the differential equation
$$ m \ddot{\vec r} = m \vec a \overset{!}{=} F = -2 k \frac 1 r \vec r \qquad\qquad (2)$$ Is this correct so far?
Alternatively with coordinates:
$$\begin{pmatrix} \ddot{x} \\ \ddot{y} \end{pmatrix} = -2 \frac k m \frac 1 {\sqrt{x^2+y^2}} \begin{pmatrix} x\\ y \end{pmatrix} \qquad \qquad(3)$$
Now my question is, how can I solve this system of differential equations?
There are some mistakes. Note that $$ {\def\p{\text{pot}}}U_\p(x,y) = \frac 12 k(x^2 + y^2) $$ Hence $$ \partial_x U_\p(x,y) = kx,\qquad \partial_y U_\p(x,y) = ky $$ there fore $$ F(x,y) = -\nabla U_\p(x,y) = -k\vec r $$ Your ode reads $$ m\ddot{\vec r} = -k\vec r $$