On page $225$ of Isaac Newton on Mathematical Certainty and Method by Niccolo Guicciardini (see here for a link), I read
In the following demonstration... Newton made a famous blunder... He wrote, "Case 1. Any rectangle as $AB$ increased by a continual motion, when the halves of the moments ${1\over 2}a$ and ${1\over 2}b$, were lacking from the sides $A$ and $B$, was $A-{1\over 2}a$ multiplied by $B-{1\over 2}b$, or $AB-{1\over 2}aB-{1\over 2}bA+{1\over 4}ab$, and as soon as the sides $A$ and $B$ have been increased by the other halves of the moments; it comes out $A+{1\over 2}a$ multiplied by $B+{1\over 2}b$, or $AB+{1\over 2}aB+{1\over 2}bA+{1\over 4}ab$. Subtract the former rectangle from this rectangle, and there will remain the excess $aB+bA$. Therefore by the total increments $a$ and $b$ of the sides, there is generated the increment $aB+bA$ of the rectangle. Q.E.D."
This reasoning seems perfectly fine to me $-$ what is the "momentous mistake"?
I'm not sure about my "interpretation" of the above discussion, but I'll try with a "long comment".
Newton did not make an "algebraic" mistake.
We have :
and :
thus :
The problem arouse in the "geometrical" interpretation [sorry, but I'm not able to draw the figures ...].
(i) If you draw the rectangle $A \times B$ inside the rectangle $(A + a) \times (B + b)$, you will have that the area of the difference between the two (the so-called gnomon) will be :
Thus, in this case, it is not true that the "total increments $a$ and $b$ of the sides [have] generated the increment $aB+bA$ of the rectangle".
(ii) But Newton did it differently.
He draw three rectangles: $A \times B$ with outside it the rectangle $(A + a/2) \times (B + b/2)$ and inside it the rectangle $(A - a/2) \times (B - b/2)$.
Now, if we compute the gnomon described by the "outer" one and the "inner" one, we have that :
I mean that we can check it "geometrically" and not only algebraically.
This is Newton's "trick": to "prove" geometrically that the term $ab$ can be discarded. He has been able to make "disappear" the unwanted second-order "infinitesimal". Of course, the key of the proof lay in its "interpretation" ...
We have to remember that he does not speak of "infinitesimal", contra Leibniz; at the same time, N's "foundations" of its fluxional calculus are at least "debatable" (like Leibniz's one), and they were debated.
See in this regard Jacob Walton's pamphlets in response to George Berkeley's The Analyst (1734); in A Vindication of Sir Isaac Newton [...] (1735) there is a long and convoluted discussion trying to justify N's proof.
See also Carl Boyer, The History of the Calculus and Its Conceptual Development (1949 - Dover reprint), pages 198-199, for comments on N's proof.