Prove that the usual (1-$\alpha$)% confidence interval for $\sigma^2$ is NOT the shortest interval. In particular, show that the minimum length interval satisfies $f_{(n+3)}(a) = f_{(n+3)}(b)$, where a and b are the endpoints of the interval, and $f_{(p)}(*)$ is the density function of a $\chi^2$ random variable with p degrees of freedom.
My approach $$P[a < \frac{(n-1)s^2}{\sigma^2} < b] = (1 - \alpha)$$ $$P[\frac{a}{(n-1)s^2} < \frac{1}{\sigma^2} < \frac{b}{(n-1)s^2}] = (1 - \alpha)$$ $$P[\frac{(n-1)s^2}{b} < \sigma^2 < \frac{(n-1)s^2}{a}] = (1 - \alpha)$$
Then I try to take the derivative of $$L(a,b) = (n-1)s^2[\frac{1}{b(a)} - \frac{1}{a}]$$ and $$\int_a^b f_{n-1} (x) dx = 1 - \alpha$$...I am struck here...Please help me move forward for this question?
We can build a Lagrangean: WE wish to minimize $ ( \frac{n-1}{q_2}S^2 , \frac{n-1}{q_1}S^2 ) $
with length: $ long( \frac{n-1}{q_2}S^2 , \frac{n-1}{q_1}S^2 )=\frac{n-1}{q_1}S^2- \frac{n-1}{q_2}S^2 $
Let's call $M =(n-1)S^2 $
So: $long( \frac{n-1}{q_2}S^2 , \frac{n-1}{q_1}S^2 )= M(\frac{1}{q_1}+\frac{1}{q_2}) $
Therefore we wish:
$ \displaystyle \min_{\{q_1,q_2\}} M(\frac{1}{q_1}+\frac{1}{q_2}) $
s.t. $ \int\limits_{q_1}^{q_2} {f_T(t)dt} = \gamma $
The latter can be expressed as $ \int\limits_{q_1}^{q_2} {f_T(t)dt} = F_T(q_2)-F_T(q_1)$
So we build the respective lagrangean $\displaystyle \min_{\{q_1,q_2\}}$ $ \mathfrak{L} $
$\mathfrak{L}= M(\frac{1}{q_1}+\frac{1}{q_2})+\lambda[\gamma-F_T(q_2)+F_T(q_1)]$
FOC
$\frac{\partial \mathfrak{L}}{\partial q_1}: \frac{-M}{q_1^2}+\lambda f_X(q_1)=0$
$\frac{\partial \mathfrak{L}}{\partial q_2}: \frac{M}{q_2^2}-\lambda f_X(q_2)=0$
Whence clearly, we get $ q_1^2 f(q_1)=q_2^2 f(q_2) $
How to interpret this?
check this: Dave Giles [2012]
Dave Giles gives us a solution based on Casella and Berger (2002) and Ferentinos and Karakostas (2007), which basically consists in "We choose the upper and lower quantiles from the distribution so as to ensure that the height to the density function is the same in each case, while still ensuring that our choice gives us the desired confidence level."
So, we have to look for $q_1$ y$q_2$ such that they have the same "height" in the density function, while naturally still covering the desired $\gamma$ expected.
Checking the book from Casella and Berger, as a matter of fact the problem is presented and certain exercises are introduced. Also, the result $f_{n+3}(q_1)=f_{n+3}(q_2)$ appears (sections 9.2 and 9.3 and page 461), though the demonstration doesn't appear.
A man named Roger Crisman studied the numerical approximation problem, and built the results on the book Journal of Undergraduate Mathematics. I attach the link where you can find it.
I know I did not answer the full question, but at least the first part on demonstrating that the typical solution is not optimal. I hope you find the solution to the second part with these hints.
REFERENCES
Dave Giles [2012] en <https://davegiles.blogspot.com/2012/02/minimizing-length-of-confidence.html>
Casella, G. and R. Berger, 2001 Statistical Inference, 2nd. ed., Duxbury.
Ferentinos, K. K. and K. X. Karakostas, 2006. More on shortest and equal tails confidence intervals. Communications in Statistics - Theory and Methods, 35, 821-829.
Crisman, Roger [1975]. Shortest Confidence Interval for the Standard Deviation of a Normal Distribution. En <https://web.archive.org/web/20160508095555/http://www.math.hope.edu/tanis/History/crisman%201975.pdf>