First-order Taylor approximation of matrix function

Question

Let $f : \Bbb C^{M \times N} \to \Bbb R_0^+$ be defined by $$ f(X) := \left\| X X^H - R \right\|_F^2 $$ I would like to find the first-order Taylor approximation of $f$.

---

I am familiar with the vector form but I cannot obtain the approximation for this function that depends on a matrix $ X $. I would like to linearize around a point $ X^0 $ to obtain something like this.

$$ f(X) = f(X^0) + \cdots $$

Update: The gradient of $ f(X) $ w.r.t. $ X \in \mathbb{C}^{M \times N} $ is

$$ \nabla_X f(X) = 2 X X^H X - R X - R^H X $$

Answer 1

$ \def\o{{\tt1}}\def\p{\partial} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\real#1{\op{\sf Real}\LR{#1}} \def\qiq{\quad\implies\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} \def\gradLR#1#2{\LR{\grad{#1}{#2}}} \def\rgrad#1#2{\frac{\p #1}{\c{\p #2}}} $For typing convenience, define the matrix variable $$\eqalign{ M = &\LR{XX^H-R} \qiq &\,dM &= X\;dX^H + dX\:\,X^H \\ &\{{\rm conjugate}\}&dM^* &= X^*dX^T + dX^*X^T \\ }$$ and the Frobenius product, which is a wonderfully concise notation for the trace $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A^*:A &= \|A\|^2_F \\ }$$ Write the objective function using the above notation.
Then calculate its differential and gradient (in the Wirtinger sense) $$\eqalign{ \phi &= M^*:M \\ &= \LR{M^*:dM} + \LR{M:dM^*} \\ &= \LR{M^*:X\;dX^H + M^*:dX\,\,X^H} + \LR{M:X^*dX^T + M:dX^*X^T} \\ &= (M^HX:dX^*) + (M^*X^*:\c{dX}) + (M^TX^*:\c{dX}) + (MX:dX^*) \\ &= \LR{M^*+M^T}X^*:\c{dX} + \LR{M+M^H}X{:dX^*} \\ \rgrad{\phi}{X} &= \LR{M^*+M^T}X^* \\ \grad{\phi}{{X^*}} &= \LR{M+M^H}X \;=\; \gradLR{\phi}{X}^* \\\\ }$$ Therefore the first-order Taylor series can be written as $$\eqalign{ \phi &= \phi(X), \quad \phi_0 = \phi(X_0) \\ \phi &= \phi_0 + \gradLR{\phi}{X}:(X-X_0) + \gradLR{\phi}{X^*}:(X-X_0)^* \\ &= \phi_0 + 2\cdot\real{\gradLR{\phi}{X}:(X-X_0)} \\\\ }$$

---

Rules for manipulating Frobenius products are easily derived. Here is a summary $$\eqalign{ A:B &= B:A &= B^T:A^T \\ CA:B &= A:C^TB &= C:BA^T \\ }$$ It is also useful to know how hermitian and complex conjugation are related $$\eqalign{ A^H &= \LR{A^T}^* &= \LR{A^*}^T \\ A^* &= \LR{A^H}^T &= \LR{A^T}^H \\ A^T &= \LR{A^*}^H &= \LR{A^H}^* \\ }$$

Answer 2

One can use a Taylor series approximation here, just as in the scalar case.

Follow the solution of Derivative of squared Frobenius norm of a matrix to compute the necessary derivative(s).

Answer 3

In general, the Taylor series of a scalar function $f：V→$, over a $$-vector space $V$ with $∈\{ℝ, ℂ\}$ can be expressed as

$$\begin{aligned} f(x+∆x) &= f(x) + ⟨f(x)∣∆x⟩_{V} + \tfrac{1}{2}⟨^2f(x)∣∆x^{⊗2}⟩_{V^{⊗2}} \\&\qquad+ \tfrac{1}{3!}⟨^3f(x)∣∆x^{⊗3}⟩_{V^{⊗3}} + … \\&= ∑_{k=0}^{∞} \tfrac{1}{k!} ⟨^kf(x)∣∆x^{⊗k}⟩_{V^{⊗k}} \end{aligned}$$

In your case $V=ℂ^n⊗ℂ^m$ and $f(x) = 4XX^HX -2RX-2R^⊤X$, hence:

$$ f(X+∆X) ≈ f(X) + ⟨4XX^HX -2RX-2R^⊤X∣∆X⟩_{ℂ^n⊗ℂ^m}$$

Note that the induced inner product on the tensor product of Hilbert spaces $⟨⋅∣⋅⟩_{ℂ^n⊗ℂ^m}$ is equivalent to the Frobenius inner product.