Let $M\simeq \Bbb{R}^{n^2}$ the set of matrices $n\times n$ and $f:M\rightarrow \Bbb{R}$ defined by $$f(X)=\det(X)$$
I need to show that the maximum of $f$ restrict to the sphere $S^{n^2-1}\subset \Bbb{R}^{n^2}$ is achieved in an orthogonal matrix, and is, in fact, $1$.
I defined $\varphi:\Bbb{R}^{n^2}\rightarrow\Bbb{R}$ by $\varphi(X)=||X||^2$. So, $\varphi^{-1}(1)=S^{n^2-1}$, and $1$ is regular value of $\varphi$, so I can use the Lagrange Multipliers Theorem.
I have
$$\textrm{grad}\,\varphi(X)=2X$$
Also, if $X=\left(x_{i,j}\right)_{1\leq i,j\leq k}$, and $X_{[i\,j]}$ is the determinant of the matrix $X$ removing the line $i$ and the column $j$, so
$$\dfrac{\partial}{\partial x_{i,j}}f(X)=(-1)^{i+j}X_{[i\, j]} $$
So, I want to solve the following system:
$$\begin{cases} \displaystyle(-1)^{i+j}X_{[i\, j]}=2\lambda x_{i,j} \\ \displaystyle\sum_{1\leq i,j\leq n}{x_{i,j}}^2=1 \end{cases} $$
I dont know what I can really do with these equations. All that I got is
$$\lambda=\dfrac{1}{2}\sqrt{\sum_{1\leq i,j\leq n}{X_{[i\, j]}}^2} $$
What can I do to solve this?
I saw this question, but its quite different. In my question, the sphere $S^{n^2-1}$ has radius $1$, but in the OP's question, the sphere has radius $n$. Then, the strategy propposed by one of the answers does not work.
Assuming you are using the Frobenius norm you have $\|A\|_F = \|U^TAU\|_F$ for all orthogonal $U$ and $\det A = \det (U^TAU)$. In particular, using the Schur decomposition, we see that the problem is equivalent to $\min_{\|T\|_F = 1, T \text{ triangular}} \det T$. It is not too hard to see that this is equivalent to $\min_{\|D\|_F = 1, D \text{ diagonal}} \det D$ and so the problem reduces to $\min_{x_1^1+\cdots + x_n^2 = 1} x_1 \cdots x_n$.
Lagrange gives (after multiplying each row by $x_k$) $x_1 \cdots x_n + 2 \lambda x_k^2 = 0$ from which we see that $|x_1|=\cdots=|x_n| = {1 \over \sqrt{n}}$ and hence $x_1 \cdots x_n = {1 \over \sqrt{n^n}}$.
This is attained for $A={1 \over \sqrt{n}}I$ (which is not orthogonal!).
Elaboration:
Lagrange gives $x_2 \cdots x_n + 2 \lambda x_1 = 0$, etc.
Multiply the first line by $x_1$, the second by $x_2$, etc. This gives $$x_1 \cdots x_n + 2 \lambda x_1^2 = 0$$ $$x_1 \cdots x_n + 2 \lambda x_2^2 = 0$$ etc, hence $2 \lambda x_2^2 = 2 \lambda x_1^2 = \cdots = 2 \lambda x_n^2$