I have used the substitution P = dy/dx to solve a first-order D.E of degree 4, so I got this:
I have to show that the above statement can be written as:
I tried to factor out first by taking p a common factor
then try to find the roots of the quadratic equation
but I never worked with such quadratic equations, any hint please how to factor the above form?
On
$$ P\left(P^3 - (x+2y+1)P^2 + (x+2y+2xy)P - 2xy\right) = 0 $$ lets rewrite as $$ P\left(P^3 - (x+2y)P^2 - P^2 + (x+2y)P + 2xyP - 2xy\right)\\ =P\left(P^2(P-1)-(x+2y)P(P-1)+ 2xy(P - 1)\right)= P(P-1)\left(P^2-(x+2y)P+ 2xy\right) =0 $$
Now $$ \left(P^2-(x+2y)P+ 2xy\right) = (P - 2y)(P - x) $$ You are done $$ P^4 - (x+2y+1)P^3 + (x+2y+2xy)P^2 - 2xyP =\\ P(P-1)(P-2y)(P-x) = 0 $$
First, be a little more careful when you "take out a common factor". $$\begin{split} P^4 - P(x+2y+1)P^3 + (x + 2&y + 2xy)P^2 - 2xyP \neq \\ & P^3 - P((x+2y+1)P^2 + (x+2y+2xy)P - 2xy). \end{split}$$ The $P^4$ term is missing entirely from the right-hand side, there's a $P^3$ term in its place, and the signs of $(x + 2y + 2xy)P^2$ and $2xyP$ have been flipped.
Since you have been given the factorization and merely need to show it's true, you can work the problem from the other end. Multiply everything out into individual terms in $x$, $y$, and $P$: $$\begin{eqnarray} P(P-1)(P-x)(P-2y) & = & P(P-1)(P^2 - xP - 2yP + 2xy) \\ & = & P(P^2(P-1) - xP(P-1) - 2yP(P-1) + 2xy(P-1)) \\ & = & P((P^3 - P^2) - (xP^2 - xP) - (2yP^2 - 2yP) + (2xyP - 2xy)) \\ & = & P(P^3 - P^2 - xP^2 + xP - 2yP^2 + 2yP + 2xyP - 2xy) \\ & = & P^4 - P^3 - xP^3 + xP^2 - 2yP^3 + 2yP^2 + 2xyP^2 - 2xyP. \end{eqnarray}$$ I'm taking more steps than I usually would here so that I can make very clear exactly where each $+$ and $-$ sign comes from. Eventually it gets easier to tell what sign each term should have so you won't need to write it in such excruciating detail, but until you get that facility, this level of detail may help.
Gather all the terms in $P^3$ together, and likewise for the terms in $P^2$, and factor out the powers of $P$: $$\begin{split} P^4 - P^3 - & xP^3 + xP^2 - 2yP^3 + 2yP^2 + 2xyP^2 - 2xyP \\ & = P^4 - P^3 - xP^3 - 2yP^3 + xP^2 + 2yP^2 + 2xyP^2 - 2xyP \\ & = P^4 + (-P^3 - xP^3 - 2yP^3) + (xP^2 + 2yP^2 + 2xyP^2) - 2xyP \\ & = P^4 - (P^3 + xP^3 + 2yP^3) + (xP^2 + 2yP^2 + 2xyP^2) - 2xyP \\ & = P^4 - (1 + x + 2y)P^3 + (x + 2y + 2xy)P^2 - 2xyP. \end{split}$$ Again, excruciating detail just to illustrate exactly why all the signs are what they are. Then, since equality is transitive, $$ P(P-1)(P-x)(P-2y) = P^4 - (1 + x + 2y)P^3 + (x + 2y + 2xy)P^2 - 2xyP. $$ Notice that $1 + x + 2y = x + 2y + 1$ and you can easily get the exact form of your original equation.