Let$$X = U(n) = \{B \in \text{GL}_n(\mathbb{C}): BB^* = I\}$$be the group of $n \times n$ unitary matrices over $\mathbb{C}$, viewed as a real manifold. Fix a diagonal $n \times n$ matrix $A$ with real entries, whose diagonal entries, call them $a_1, \dots, a_n$, are pairwise distinct. (Of course, usually, $A \notin U(n)$.) Define a (smooth) function$$f: U(n) \to \mathbb{R},\,B \mapsto \text{Re}(\text{Tr}(AB)),$$where $\text{Re}(z)$ is the real part of a complex number $z$.
- What are the critical points of $f$?
- What is the second differential of $f$ at each of these points?
Let $A=diag(a_j),B=[b_{j,k}]=[c_{j,k}]+i[d_{j,k}]$; then $f(B)=Re(\sum_ja_jb_{j,j})=\sum_ja_jc_{j,j}$. Clearly, for every $j$, $|c_{j,j}|\leq 1$. Conversely, consider $(u_j)_j$ s.t., for every $j$, $|u_j|\leq 1$; then we can write $u_j=\cos(\theta_j)$ and $diag(e^{i\theta_j})$ is unitary. We assume that $a_1,\cdots,a_p>0,a_{p+1},\cdots,a_q<0$ and the other $(a_j)_j$ are zero.
Finally, we consider the function $g:(c_{j,j})_j\in [-1,1]^{p+q} \rightarrow \sum_ja_jc_{j,j}$. The local extrema, that are, global maximum and minimum, are reached for $(c_{j,j})=(signum(a_j))$ and $(c_{j,j})=(-signum(a_j))$.
EDIT 1. The extreme points $(\pm 1)_j$ of $[-1,1]^{p+q}$, the convex domain of $g$, give birth to the following critical points of $f$: the diagonal matrices $diag(\pm 1,\cdots,\pm 1)$.
The second derivative is useless because we know where are the local extrema.
EDIT 2. I don't see what is the interest of studying the critical points of $f$. Or John offers us a homework, either he thinks that it is the best method to study the extrema of $f$ (obviously, he is wrong !). In fact, when I replace $f$ with $g$, I kill the critical points of $f$.
Consider the case $n=2$. $(\theta ,a,b,h)\in \mathbb{R}^4\rightarrow \begin{pmatrix}\cos(\theta)e^{ia}&-\sin(\theta)e^{i(a+h)}\\\sin(\theta)e^{ib}&\cos(\theta)e^{i(b+h)}\end{pmatrix}$ is a local parametrization of $U(2)$ (CORRECTION) only when $J=\sin(\theta)\cos(\theta)\not= 0$.
Here, we assume that $a_1,a_2\not= 0$ and $f:(\theta ,a,b,h)\in \mathbb{R}^4\rightarrow \cos(\theta)(a_1\cos(a)+a_2\cos(b+h))$. Its critical points (for $J\not= 0$) are solutions of $a_1\cos(a)+a_2\cos(b+h)=0,\sin(a)=0,\sin(b+h)=0$. If $a_1\not=-a_2$, then no such solutions; if $a_1=-a_2$, then $\cos(a)=\cos(b+h)=\pm 1$, that is, an infinity of solutions that depend on the parameters $\theta,h$.