My attempt is the following.
Using polar coordinates, let $(x, y) = (r \cos \theta, r \sin \theta)$, where $$r^2 = \frac{\frac{25}{4}5}{25 \cos^2\theta+\frac{25}{4} \sin^2\theta}.$$ Then, the expression $x^2+12xy+2y^2$ becomes $r^2(\cos^2\theta+12 \cos\theta \sin\theta+2\sin^2\theta)$. It's a function of $\theta$, so find the derivative $f'$ of $f$, and then, let $f'$ equal $0$ to find the critical point.
But I get a really messy $f'$, and I don't know where to go from here.
On
The extrema are reached with the two curves $f(x,y)=x^2+12xy+2y^2$ and $g(x,y)=4x^2+y^2$ tangential to each other, i.e. $\frac{f_x’ }{f_y’ }= \frac{g_x’ }{g_y’ } $, or
$$ \frac{x+6y}{6x+2y}= \frac{4x}y\implies (2y+3x)(3y-8x)=0$$
which leads to $y= -\frac32x,\> y= \frac83x$. Substitute them into $g(x,y)=25$ to get the tangential points $(\pm\frac32, \pm4)$ and $(\pm2, \mp3)$. Plug them into $f(x,y)$ to obtain
$$f_{max} = f( \pm\frac32, \pm4 ) =\frac{425}4 , \>\>\>\>\> f_{min} = f( \pm2, \mp3) =-50$$
Considering that $(2x)^2 + y^2 = 4x^2 + y^2 = 25,$ we have that $2x = 5 \cos \theta$ and $y = 5 \sin \theta$ by the usual polar coordinates identifications. Consequently, we have that $$f(x, y) = x^2 + 12xy + 2y^2 = \frac{25}{4} \cos^2 \theta + 12 \cdot \frac{5}{2} \cos \theta \cdot 5 \sin \theta + 2 \cdot 25 \sin^2 \theta = g(\theta),$$ and we can use the Extreme Value Theorem to find the absolute maximum of $g(\theta)$ on the closed interval $[0, 2 \pi].$ Using the Product Rule and the Chain Rule, we have that $$g'(\theta) = -\frac{25}{2} \sin \theta \cos \theta - 150 \sin^2 \theta + 150 \cos^2 \theta + 100 \sin \theta \cos \theta = 150 u^2 + \frac{175}{2} uv - 150 v^2$$ by the substitution $u = \cos \theta$ and $v = \sin \theta.$ Observe that we have $g'(\theta) = 0$ if and only if $12 u^2 + 7uv - 12 v^2 = 0$ if and only if $(4u - 3v)(3u + 4v) = 0$ if and only if $\frac u v = \frac 3 4$ or $\frac u v = \frac{-4} 3$ if and only if $\cot \theta = \frac 3 4$ or $\cot \theta = \frac{-4} 3$ if and only if $\tan \theta = \frac 4 3$ or $\tan \theta = \frac{-3} 4.$
Can you finish the problem? (Use the Extreme Value Theorem on the following four points: the two critical points arising from $g'(\theta) = 0$ for $0 < \theta < 2 \pi$ and the two endpoints $\theta = 0$ and $\theta = 2 \pi.$)