Consider $\begin{bmatrix} \dot x\\ \dot y \end{bmatrix} =\begin{bmatrix} xy^2 -xy\\ x-y \end{bmatrix}$. If we take the Jacobian and evaluate it at $(0,0)$, one of the eigenvalues is $-1$ and the other is $0$. If this were a linear system, we would know that the solution would converge parallel to the eigenvector associated to $-1$. But, as long this is non-linear, now should try to use H-G theorem. But one cannot use it if the real part of eigenvalues is zero. In this case, when only one of them has a zero real part, can I apply it?
Thanks a lot!
You don't give enough information to get an answer. It depends on what you want:
Do you want to determine the type of stability of the origin?
If so, then you can use the center manifold theorem plus an appropriate version of the Grobman-Hartman theorem using local coordinates given by the stable, unstable and center manifolds (which are transverse, of course without unstable manifold in your example), to show that the system is locally topologically equivalent to a system of the form $u'=-u$, $v'=f(v)$ for some function $f$ to be determined (as usual). The stability of the origin in the original systems is then given by the stability of $v'=f(v)$.
PS. Why do you write "Hartman-Grobman" when both proved the theorem independently and more or less at the same time? It should be "Grobman-Hartman theorem", if we really don't want to mention the independent "modern" proofs by Palis and by Pugh.
Do you want to obtain a conjugacy to a linear system near the origin?
This is not possible in general, one cannot know without making the computations (sometimes it is, sometimes it is not). What is true in general is (for example as a consequence of the approach described above, but there are other methods) that you can introduce coordinates such that the nonlinear parts along the stable and unstable directions can be erased with a local conjugacy.