Directional derivative of l2 norm

Question

I have a function,

$Q(b) = 1/2||y-Xb||_2^2$,

and want to find directional derivative, defined as

$\lim_{t\to 0}(f(b+tu)-f(b))/t$.

I am having difficulty showing that

$Q(\hat{b} + tu) - Q(\hat{b}) = -t<y-Xb, Xu> + o(|t|)$,

and then that the directional derivative of $Q$,

$D_Q(\hat{b},u) = -\sum_{j\in[p]}u_jx_j^T(y-X\hat{b}).$

Answer 1

Use the fact that

$$ ||x||_2^2 = \langle x | x \rangle $$

And then if all the quantities are real

\begin{eqnarray} Q(b + t u) &=& \frac{1}{2}|| y - X(b + tu) ||^2_2 = \frac{1}{2}\langle y - X(b + tu)|y - X(b + tu)\rangle \\ &=&\frac{1}{2} \langle y - Xb - tXu | y - Xb - tXu \rangle \\ &=&\frac{1}{2}\langle y - Xb | y - Xb \rangle -t \langle y - Xb | Xu\rangle + \frac{1}{2}t^2\langle Xu | Xu\rangle \\ &=& Q(b) -t \langle y - Xb | Xu\rangle + \frac{1}{2}t^2\langle Xu | Xu\rangle \end{eqnarray}

The directional derivative is then

\begin{eqnarray} D_Q(b, u) &=& \lim_{t\to 0} \frac{Q(b + tu) - Q(b)}{t} = \lim_{t\to 0}\left[ - \langle y - Xb | Xu\rangle + \frac{1}{2}t\langle Xu | Xu\rangle \right] \\ &=& - \langle y - Xb | Xu\rangle \end{eqnarray}

Answer 2

For convenience, define the vector $$z=Xb-y$$Write the function in terms of the dot product and this new variable, then calculate the differential, and keep changing the independent variable until you have an expression in terms of $dt$ $$\eqalign{ Q &= \frac{1}{2}\,z\cdot z \cr\cr dQ &= z\cdot dz \cr &= z\cdot X\,db \cr &= z\cdot Xu\,dt \cr &= (Xb-y)\cdot Xu\,dt \cr\cr }$$ From which the derivative is seen to be $$\eqalign{ \frac{dQ}{dt} &= (Xb-y)\cdot Xu }$$