How to find maximum of the function $u(x,y)=x^2-y^2+x^2y^2$ ,$(x,y)\in \overline{B_2}(0) $ (closed ball of radius $2$)
The unique critical point is $(0,0)$ but is not $max(u)$. I've tried to use polar coordinates but without result.
Ok second part...Let's say we found $max(u)=(x_0,y_0)$. We have to calculate $\cfrac{\partial u}{\partial \nu}(x_0,y_0)$ (exterior normal derivative of $u $ in $(x_0,y_0)$ ) and this is $\nabla u(x_0,y_0)\cdot\nu=(2x_0+2y_0^2 x_0,-2y_0+2x_0^2 y_0) \cdot\nu(x_0,y_0)$ but how to fin $\nu.$ It would be useful for me to see how do we find $\nu.$
You have $$\nabla u(x,y)=\bigl(2x(y^2+1),2y(x^2-1)\bigr)\ ,\tag{1}$$ and it is easy to check that $(0,0)$ is the only critical point in the interior of $B_2$, and $u(0,0)=0$. In order to deal with the boundary $\partial B_2$ we use the parametrization $$\partial B_2:\quad\phi\mapsto\bigl(x(\phi),y(\phi)\bigr)=(2\cos\phi,2\sin\phi)\qquad\bigl(\phi\in{\mathbb R}/(2\pi)\bigr)$$ and then have to analyze the pullback $$f(\phi):=u\bigl(x(\phi),y(\phi)\bigr)=4\cos^2\phi-4\sin^2\phi+16\cos^2\phi\sin^2\phi\ .$$ We obtain $$f(\phi) =24\cos^2\phi-16\cos^4\phi-4\tag{2}$$ and then $$f'(\phi)=-16\cos\phi\sin\phi(3-4\cos^2\phi)\ .$$ There are eight zeros of $f'$, coming from the equations $\cos\phi=0$, $\>\sin\phi=0$, $\>\cos\phi=\pm{\sqrt{3}\over2}$. This means that the function $f$ defined in $(2)$ has eight critical points. You have to compute the value of $f$ in all of these points, and to select the largest of these values, if it is $>0$. Otherwise the maximum of $u$ is at the origin.
It seems that, e.g., for $\phi={\pi\over6}$ we have a maximum. Hence we consider the point $$(x_0,y_0):=\bigl(2\cos{\pi\over6},2\sin{\pi\over6}\bigr)=\bigl(\sqrt{3},1\bigr)\ .$$ Using $(1)$ we find $$\nabla u(x_0,y_0)=\bigl(4\sqrt{3},4\bigr)\ .$$ Since $\partial B_2$ is a circle centered at the origin the unit normal $\nu$ at $(x_0,y_0)$ is nothing else but $$\nu=\left(\cos{\pi\over6},\sin{\pi\over6}\right)=\left({\sqrt{3}\over2},{1\over2}\right)\ .$$ Therefore we obtain $${\partial u\over\partial \nu}(x_0,y_0)=\langle\,\nabla u(x_0,y_0), \>\nu\,\rangle=4\sqrt{3}\cdot {\sqrt{3}\over2}+4\cdot {1\over2}=8\ .$$