In cartesian coordinates, $dA = dx\,dy$ . As $x=\rho \cos\phi\ $ and $y=\rho \sin \phi \ $,
then: $dx= \cos\phi\, d\rho - \rho \sin\phi\, d\phi$ and $dy = \sin\phi\, d\rho + \rho \cos\phi \, d \phi $.
So $dx\, dy = \rho\, d\rho\, d\phi(\cos^2\phi-\sin^2\phi) $, (ignoring second order terms in $d\rho$ and $d\phi$)
Why is this not equal to the correct result of $dA=\rho\, d\rho\, d\phi$?
Because $d\rho\, d\phi=-d\phi\, d\rho$