Let $g : \mathcal{M}_n(\mathbb{R}) \times \mathcal{M}_n(\mathbb{R}) \rightarrow \mathcal{M}_n(\mathbb{R})$ such that $$g(A,B) = (A^T)BA $$ where $(A^T)$ is the transpose matrix of A. Show that $g$ is $\mathcal{C}^1$ and calculate its differential function.
So I started of first by studying its partial derivatives. We have:
$$\frac{\partial g}{\partial A}(A,B) = B, \frac{\partial g}{\partial B}(A,B) = (A^T)A $$
I am unsure if I calculated them correctly, not too sure when I use matrices. They seem to be continuous(not sure how to justify that), so $g$ is $\mathcal{C}^1$.
Now, when I try to calculate the differential in two different methods, I realise I might have done a mistake.
I could simply state that $$dg_{A,B}(H,K) = \frac{\partial g}{\partial A}(A,B)H + \frac{\partial g}{\partial B}(A,B)K = BH + (A^T)AK $$
Yet, if I calculate when $t \not =0$, $t \rightarrow 0$ the fraction:
$$\frac{1}{t}(g((A,B)+(tH,tK)) - g(A,B)) = \frac{g(A+tH, B+TK) - g(A,B)}{t} = (A^T)KA + (H^TBA) + (A^T)BH$$
Which is different from what I found with the partial derivatives. I don't know where is my mistake.
Here's an approach with Fréchet derivative. Note that $$\begin{align} g((A,B)+(H_1,H_2))=A^TBA&+A^TBH_1+A^TH_2A+H_1^TBA\\ &+H_1^TBH_1+H_1^TH_2A+H_1^TH_2H_1+A^TH_2H_1 \end{align}$$
The candidate for the differential of $g$ at $(A,B)$ is $(H_1,H_2)\mapsto A^TBH_1+A^TH_2A+H_1^TBA$ which is clearly linear (and continuous because of finite dimension).
Consider $\|\cdot \|$ a submultiplicative norm on $\mathcal{M}_n(\mathbb{R})$ and set a norm on $\mathcal{M}_n(\mathbb{R}) \times \mathcal{M}_n(\mathbb{R})$ by letting $\|(A,B)\|=\max(\|A\|, \|B\|)$. Since the trace function is continuous, there exists some $\alpha$ such that $\frac{\|H_1^T\|}{\|H_1\|}\leq \alpha$.
Note for example that $\frac{\|H_1^TBH_1\|}{\|(H_1,H_2)\|}\leq \frac{\alpha\|H_1\|^2\|B\|}{\|H_1\|}$ and derive similar bounds for the other summands.