I Tried letting the radius of the cone be "$r$" and the height "$h$" and then the radius of the hemisphere "$R$" is $\sqrt{(r²+h²)}$.
I tried letting an angle $\theta$ and tried to simplify things down, and taking the derivative, but it just isn't working, is there some other nice approach because I am not finding a similar problem anywhere on the internet - I mean, it seemed a little too trivial to me at first, but since the cone is inverted so I just couldn't let $R=h$ and substitute the values in, and the cone is also in a hemisphere not a sphere, so I couldn't really figure out what to do.
An explanation would be very much appreciated.
You want to maximum the volume
$V = \frac{1}{3} r^2 h $
where $r, h$ satisfy
$ r^2 + h^2 = R^2 $
and $R$ is the radius of the sphere.
From this, you can write $r^2 = R^2 - h^2$ and substitute in the volume function
$ V = \frac{1}{3} h (R^2 - h^2) $
Now differentiate with respect to $h$,
$ \dfrac{dV}{dh} = \frac{1}{3} ( R^2 - h^2 - 2 h^2 ) = \frac{1}{3} (R^2 - 3 h^2) $
The second derivative is $ \dfrac{d^2 V}{d h^2} = -2 h $
Setting $\dfrac{dV}{dh} = 0 $ gives us,
$ h^2 = \frac{1}{3} R^2 $
So that
$ h = \dfrac{1}{\sqrt{3}} R $
To check that this actually a maximum, substitute into the expression of the second derivative, you'll get a negative value, indicating that it is a maximum.