Write this in matrix form?

Question

I'm trying to work on my linear algebra skills. I came across this equation in Bishop PRML (Equation 4.93):

$$ \nabla E(w) = \sum_{n=1}^N(\mathbf{w}^T\mathbf{\phi}_n-t_n)\mathbf{\phi}_n = \mathbf{\Phi}^T\mathbf{\Phi w} - \mathbf{\Phi^T t}$$

Where $\mathbf{w, \phi_n, t}$ are vectors and $\mathbf{\Phi}$ is a $N$ by $M$ matrix whose $n^{th}$ row is given by $\mathbf{\phi}_n^T$.

The matrix form confuses me. I understand how he derives the $\mathbf{\Phi ^Tt}$ part, but really can't figure out how he came up with $\mathbf{\Phi^T\Phi w}$

I hope one of you can explain it to me :)

Thanks

P.S. Tips on how to structurally figure these matrix forms out in general are really appreciated.

Answer 1

\begin{equation} \Phi = \begin{bmatrix} \phi_1^T \\ \vdots \\ \phi_N^T \end{bmatrix} \end{equation} So \begin{equation} \Phi w = \begin{bmatrix} \phi_1^T \\ \vdots \\ \phi_N^T \end{bmatrix} w = \begin{bmatrix} \phi_1^T w\\ \vdots \\ \phi_N^Tw \end{bmatrix} \end{equation} Therefore \begin{equation} \Phi^T\Phi w = \underbrace{ \begin{bmatrix} \phi_1 & \ldots & \phi_N \end{bmatrix}}_{\Phi^T} \underbrace{ \begin{bmatrix} \phi_1^T w\\ \vdots \\ \phi_N^Tw \end{bmatrix}}_{\Phi w} = \sum\limits_{n=1}^N \phi_n (\phi_n^T w) \tag{1} \end{equation} Also \begin{equation} \Phi^T t = \begin{bmatrix} \phi_1 & \ldots & \phi_N \end{bmatrix} \begin{bmatrix} t_1 \\ \vdots \\ t_N \end{bmatrix} = \sum\limits_{n=1}^N t_n \phi_n\tag{2} \end{equation} Combining $(1)$ and $(2)$, we get \begin{equation} \Phi^T\Phi w-\Phi^T t = \sum\limits_{n=1}^N \phi_n (\phi_n^T w) - \sum\limits_{n=1}^N t_n \phi_n = \sum\limits_{n=1}^N\phi_n (\phi_n^T w) - t_n\phi_n \end{equation} Since $(\phi_n^T w)$ and $t_n$ are scalars, we could flip \begin{equation} \Phi^T\Phi w-\Phi^T t = \sum\limits_{n=1}^N (\phi_n^T w)\phi_n - t_n\phi_n = \sum\limits_{n=1}^N [(\phi_n^T w)- t_n]\phi_n \end{equation}

Answer 2

Use the standard vector basis $\{e_n\}$ to express the subscripted quantities. $$\eqalign{ \phi_n &= \Phi^Te_n &\implies \phi_n^T = e_n^T\Phi \cr t_n &= e_n^Tt &\implies t = t_ne_n \cr }$$ Recall that the identity matrix can be written in terms of the basis vectors. $${\mathbb I}=\sum_n e_ne_n^T = e_ne_n^T$$ where the far RHS uses the Einstein summation convention, i.e. omits the explicit $\Sigma$-symbol and implicitly sums over the repeated index. It removes the visual "clutter" from the equations, so I'll use it below.

Substitute the basis vectors into your equation $$\eqalign{ (w^T\phi_n-t_n)\phi_n &= \phi_n\phi_n^Tw-\phi_nt_n \cr &= (\Phi^Te_n)(e_n^T\Phi)\,w-(\Phi^Te_n)\,t_n \cr &= \Phi^T{\mathbb I}\,\Phi w-\Phi^T\,t \cr &= \Phi^T\Phi w-\Phi^Tt \cr }$$