$r$ - radius of base
$h$ - height of cone
$l$ - slant height of cone
$V$ - Volume of cone
$A$ - surface area of cone
First of all I myself don't know what's the meaning of phrase "given surface area", since it is written given I would assume it to be constant. I need to maximize the volume: $$ V = \frac{\pi r^2 h}{3}, $$ $$ \frac{dV}{dr} = \frac {\pi}{3} \left(2rh + r^2 \frac{dh}{dr}\right). $$
To obtain critical points set $dV/dr = 0$: $$ 2rh + r^2 \frac{dh}{dr} = 0, $$ $$ \frac{dh}{dr} = -2\frac{h}{r}. $$ Now, $A = \pi r \left( r + \sqrt{h^2 + r^2} \right)$. Using $dA/dr = 0$ and $dh/dr = -2h/r$ we would get $$ h = \sqrt{8}r, $$ I have got the answer but how can I check whether it's a minimum volume or maximum volume.
For example if I put $h = \sqrt{8}r$ in the equation of volume we would get
$V = (\sqrt{8}/3) \pi r^3$, so if I double differentiate it
$d^2V/dr^2 = 2\cdot \sqrt{8} \cdot \pi \cdot r^2 $, no matter what value or $r$ I put I always get a positive value which means $h = \sqrt{8}r$ represents a minimum volume.
Please spot out my mistake as I'm myself unable to do it.
you seek an extremum of the function $$ f(r,h,l) = r^2h $$
subject to the imposed condition: $$ r^2 + rl = c $$ and the geometrical relation $$ h^2 + r^2 - l^2 = 0 $$
form the objective function: $$ L = r^2h + \lambda(r^2 + rl) + \mu(h^2 + r^2 - l^2) $$
for an extremum $$ \frac{\partial L}{\partial h} = r^2 + 2 \mu h = 0 \\ \frac{\partial L}{\partial r} = 2rh + \lambda(2r+l) + 2 \mu r = 0 \\ \frac{\partial L}{\partial l} = \lambda r - 2 l \mu = 0 $$ from which $$ 2 \mu = - \frac{r^2}h = \frac{\lambda r}l $$ giving $$ \lambda = 2\mu \frac{l}r $$ now, substituting in $(1)$ $$ 2h + 2\mu \left( 2l + \frac{l^2}r + r \right) = 0 $$ giving $$ 2h^2 = 2lr +r^2 + l^2 = 2lr + r^2 + r^2+h^2 $$ so, finally $$ h^2 = 2(lr + r^2) = \frac{2A}{\pi} $$