I am stuck on the following two questions on matrix Algebra:
Let $x$ be a vector. Find the following:
- $\frac{\partial}{\partial x}||x\otimes x||^2$, here ||.|| denotes Euclidean norm of a vector and $\otimes$ denotes the kroneckar product.
- $\frac{\partial}{\partial x^T}(\frac{x}{||x||})$.
My try: (i) Let us take $x=(x_1,x_2,\dots,x_n)$. Then $x\otimes x=\begin{pmatrix}x_1^2 &x_1x_2& x_1x_3 &\dots & x_1x_n \\ x_2x_1 &x_2^2& x_2x_3 &\dots & x_2x_n \\ \dots & \dots& \dots & \dots & \dots\\ \dots & \dots& \dots & \dots & \dots\\ x_nx_1 &x_nx_2& x_nx_3 &\dots & x_n^2 \end{pmatrix}$
I am confused about how do I take $||x\otimes x||$ and find its derivative in terms of $x$.
(II) Since $||x||$ is a scalar, so we can write $\frac{\partial}{\partial x^T}(\frac{x}{||x||})=\frac{1}{||x||}\frac{\partial}{\partial x^T}(x)$. Here I am stuck how to find the derivative of $x$ with respect to $x^T$.
Can someone please help me to clear my doubts and complete the above problems?
$ \def\a{\alpha}\def\b{\beta}\def\l{\lambda}\def\o{{\tt1}} \def\B{\Big}\def\L{\left}\def\R{\right} \def\LR#1{\L(#1\R)} \def\BR#1{\B(#1\B)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\v#1{\operatorname{vec}\LR{#1}} \def\qiq{\quad\implies\quad} \def\t{\otimes} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} $The Frobenius product is a concise notation for the trace $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \big\|A\big\|^2_F \\ }$$ This is also called the double-dot or double contraction product.
When applied to vectors $(n=\o)$ it reduces to the standard dot product.
The distributive rule for a mixed Kronecker-Frobenius product is $$\eqalign{ (A\t B):(X\t Y) = (A:X)\t(B:Y) \\ }$$
First calculate the gradient of the simple function $$\eqalign{ \a &= \|x\|^2 = x:x \qiq \c{d\a = 2x:dx} \\ }$$ Then calculate the differential (and gradient) for the first function of interest $(\b)$. $$\eqalign{ \b &= \|x\t x\|^2 \\&= (x\t x):(x\t x) \\&= (x:x)\t(x:x) \\&= \a^2 \\ d\b &= 2\a\;\c{d\a} \\&= 2\a\,\LR{2x\,dx} \\&= 4\|x\|^2\,x\,dx \\ \grad{\b}{x} &= 4\|x\|^2\,x \\ }$$ The second function of interest $(y)$ can also make use of the simple $\a$ function. $$\eqalign{ \l &= \|x\| \\ \l^2 &= \a \qiq 2\l\,d\l = d\a = 2x^Tdx \\ d\l &= \l^{-1}x^Tdx \\ \\ y &= \frac{x}{\|x\|} = \l^{-1}x \\ dy &= \l^{-1}dx-x\l^{-2}\,\c{d\l} \\ &= \l^{-1}I\,dx-x\l^{-2}\,\c{\l^{-1}x^Tdx} \\ &= \l^{-1}\LR{I-yy^T}\,dx \\ \grad{y}{x} &= \l^{-1}\LR{I-yy^T} \\\\ }$$
NB: Your expansion of $(x\t x)$ is incorrect. What you have written is actually the matrix $xx^T$ which you must vectorize to recover the desired Kronecker product, i.e. $$(x\t x) = \v{xx^T}$$ However, there is another property of the Frobenius product $$\v{A}:\v{A} = A:A$$ which allows you to use your matrix without vectorizing it $$\eqalign{ \b &= \v{xx^T}:\v{xx^T} \\&= \LR{xx^T}:\LR{xx^T} \\ &= \trace{xx^Txx^T} \\&= \a^2 }$$