Prove if $Q(x,y)=Ax^2+Bxy+Cy^2$ is a quadratic form, then the constant $M = |A|+|B|+|C|$ satisfies the property that $|Q(a,b)| \leqslant M \cdot \max(|a|^2,|b|^2)$ for every $a,b \in \mathbb{R}.$
We have $$|Aa^2+Bab+Cb^2| \leqslant \left(|A|+|B|+|C| \right) \cdot |a|^2 = |A| \cdot |a|^2 + |B| \cdot |a|^2 + |C| \cdot |a|^2$$ if $|a|^2 > |b|^2$ and $$|Aa^2+Bab+Cb^2| \leqslant \left(|A|+|B|+|C| \right) \cdot |b|^2 = |A| \cdot |b|^2 + |B| \cdot |b|^2 + |C| \cdot |b|^2$$ if $|b|^2 > |a|^2,$ but how do we progress from here?
You are almost right.
If $|a|>|b|$, $|Q(a,b)|\leqslant M|a|^2=M\max(|a|^2,|b|^2)$
If $|a|<|b|$, $|Q(a,b)|\leqslant M|b|^2=M\max(|a|^2,|b|^2)$