This was a two part question;
Part 1)
"Given f(x,y) = 36(y-x^2)^2+(x-1)^2, show that the hessian is positive semi-definite in the region y <= x^2 + 1/72."
I solved this I think by subbing y <= x^2 + 1/72 into f(x,y) to get;
f(x, x^2 + 1/72) <= x^2 - 2x + 145/144 (I cut out all the algebra, this was the final result)
then solving for the hessian (which was [2,0;0,0] and I think because there are no variables, its just an equality instead of inequality now), giving eigenvalues 2 and 0. This indicates positive semi-definiteness, which should indicate convexity of some form I believe. I'd appreciate some comment on this 'proof',but I do think it shows that the region is positive semi-definite. But the second part asks;
Part 2)
" Explain why the function is not convex in this region? " and I have no idea. To my knowledge, positive semi-definite means convexity. I would really really appreciate some form of explanation here. Thank you very much!
The Hessian of your function $f$ is the matrix of its second partial derivatives: $$H(x,y):=\left[\matrix{f_{xx}&f_{xy}\cr f_{xy}&f_{yy}\cr}\right]=\left[\matrix{2+432x^2-144y&\quad-144x\cr -144x&\quad72\cr}\right]\ .$$ The determinant of this matrix is $$\det H(x,y)=144(1+72x^2-72y)\ .$$ This determinant is positive in the region $\Omega: \ y< x^2+{1\over72}$. As $f_{yy}>0$ everywhere this allows to conclude that the quadratic form associated to $H$ is positive definite in all points of $\Omega$. This implies that $f$ is convex in every convex subregion of $\Omega$. It is however not convex in the full region $\Omega$ since $\Omega$ is not convex. E.g., the segment connecting the two points $(\pm1,1)\in\Omega$ contains the point $(0,1)\notin\Omega$.