Consider the initial value problem:
$\frac{dy}{dx}$= $xy - x^2 + 1$
with $y(0) = 0$
In order the find the unique interval we first find that $f(x,y)$ and $f_y$ are continuous in the rectangle:
$R: \{(x,y): 0 < x < 0 + a, -b < y < b\}$
next we find the $M = Max \left| f(x,y) \right|$.
Due to this being a past paper question, I have that $M = ab + 1$. But I cannot figure out how this answer was derived and how generally $M$ should be found for other similar questions.
Assuming that there is no error in your question, the bound that you give is not correct. Choose $a=10, b=20$ and note that $\lim \limits _{(x,y) \to (10,-20)} |f(x,y)|=|f(10,-20)|=299 \nleq 10 \cdot 20 +1=201$.
Ont the other hand, you have $|f(x,y)| = |xy-x^2+1| \leq |xy| + |x^2| + |1| \leq ab + a^2 +1 = a(a+b)+1$.