$\frac{dy}{dx} = \frac{x^{2}y}{x^{3}+xy+y^2}$ is invariant under $(x,y) \mapsto (\frac{x}{1+\varepsilon y},\frac{y}{1+\varepsilon y})$
I can't make both sides equal when I have a variable depends on two variables I use $\frac{\mathrm{d}y}{\mathrm{d}x} = \frac{\mathrm{D}y}{\mathrm{D}x}$ ($D=$ total derivative), but I can't make both equal!! How to do that?
I want to mention that this example is arbitrary I can not make any invariant when any variable equal any combination of $x$ and $y$.
Consider a transformation group on $\mathbb{R}^2$, i.e. an application $(x,y)\to(\bar{x},\bar{y})$. Also assume that $x,y$ are related by an ODE: $$\dfrac{d y}{d x} =f(x,y)$$
Now, the application of the chain rule yields: $$ \begin{align} \frac{D \bar{x}}{Dx}&=\frac{\partial \bar{x}}{\partial x} + \frac{\partial \bar{x}}{\partial y} \frac{d y}{d x} = \frac{\partial \bar{x}}{\partial x} + \frac{\partial \bar{x}}{\partial y} \,f(x,y) \\ \frac{D \bar{y}}{Dx}&=\frac{\partial \bar{y}}{\partial x} + \frac{\partial \bar{y}}{\partial y} \frac{d y}{d x} = \frac{\partial \bar{y}}{\partial x} + \frac{\partial \bar{y}}{\partial y} \,f(x,y) \end{align} $$ Again by the chain rule, you can consider how the group acts on the ODE: $$\dfrac{d \bar y}{d \bar x} =\frac{\dfrac{D \bar{y}}{Dx}}{\dfrac{D \bar{x}}{Dx}}$$ and then check if $$\dfrac{d \bar y}{d \bar x}=f(\bar x,\bar y)$$
In your example, straightforward (but tedious) calculation shows: $$ \dfrac{d \bar y}{d \bar x}=\frac{x^2 y}{x y (\epsilon y+1)+y^2 (\epsilon y+1)+x^3} $$ whereas $$ f(\bar x,\bar y)=f\left(\frac{x}{\epsilon y+1},\frac{y}{\epsilon y+1}\right)=\frac{x^2 y}{x y (\epsilon y+1)+y^2 (\epsilon y+1)+x^3} $$ so the ODE is invariant under the group action.