I saw this question in a mock exam:
Given positive real constants $a, b$ and $f(x,y)=(a x)^2+ abxy +(by)^2-a^2 x-b^2 y$, maximize $f$ over the interior and boundary of the convex hull of $(0,0), (1,0), (0,1)$.
What is the shortest way to answer this problem when sitting on a maths exam?
I don't think there are useful shortcuts here.
Check the inside
This is the usual optimization procedure where you are looking for stationary points satisfying $f_x(x,y) = 0 = f_y(x,y)$ and classifying them to get a maximum.
Check the boundary
Formally you could do Lagrange multipliers but since you are on one of 3 lines, 2 with easy form, this is easier by hand:
Can you finish?