$g(\theta):= f(\theta {\bf y} + (1 − \theta){\bf x})$, find $g'$.

Question

Let $f: \mathbb{R}^n \longrightarrow \mathbb{R}$ and let $g: \mathbb{R} \longrightarrow \mathbb{R}$ given by $g(\theta):= f(\theta {\bf y} + (1 − \theta){\bf x})$.

I want to calculate the derivative $g'$.

I set $z=\theta {\bf y}+ (1 − \theta){\bf x}$, then:

$$ \frac{\mathrm{d}g}{\mathrm{d}\theta}= \frac{\mathrm{d}f}{\mathrm{d}\theta}= \frac{\mathrm{d}f}{\mathrm{d}z} \frac{\mathrm{d}z}{\mathrm{d}\theta}= \nabla f^\mathsf{T} ({\bf y}-{\bf x}) $$

Is this formally correct? If it is not wrong, is it possible to write it better?

Answer 1

Yes what you are doing is correct. Perhaps if you want to add more steps, note that

$$ z_{i}=\theta y_{i}+\left(1-\theta\right)x_{i} $$

so that

$$ \frac{\partial z_{i}}{\partial\theta}=y_{i}-x_{i}. $$

Then,

$$ \frac{dg}{d\theta}=\frac{\partial f}{\partial z_{1}}\frac{\partial z_{1}}{\partial\theta}+\cdots+\frac{\partial f}{\partial z_{n}}\frac{\partial z_{n}}{\partial\theta}=\frac{\partial f}{\partial z_{1}}\left(y_{1}-x_{1}\right)+\cdots+\frac{\partial f}{\partial z_{n}}\left(y_{n}-x_{n}\right), $$

which is the inner product you had above. Note that $\nabla f$ is evaluated at $z$.

Answer 2

Take ${\bf x}=(x_1,x_2,\dots,x_n),{\bf y}=(y_1,y_2,\dots,y_n)$. Then you have $$g(\theta)=f(\theta y_1+(1-\theta)x_1,\theta y_2+(1-\theta)x_2,\dots,\theta y_n+(1-\theta)x_n)$$

Note that you have a something of the form $$g(\theta)=f(H_1(\theta),H_2(\theta),\dots,H_n(\theta))$$

All you have to do is apply the chain rule.

In particular, if we have $F:\Bbb R\to \Bbb R^n$ with $$F(\theta)=(H_1(\theta),H_2(\theta),\dots,H_n(\theta))$$

then $$F'(\theta)=(H_1'(\theta),H_2'(\theta),\dots,H_n'(\theta))$$

and $$ (f\circ F)'(\theta)=\nabla f( F(\theta))\cdot F'(\theta)$$

Now, as you correctly noted $$H_n(\theta)=\theta y_n+(1-\theta)x_n$$ is such that $$H_n'(\theta)=y_n-x_n$$ so that $$F'(\theta)={\bf y-x}$$

So you ultimately obtain $$g'(\theta)=\nabla{\bf f}(\theta{\bf y}+(1-\theta){\bf x})\cdot ({\bf x-y})=\nabla{\bf f}({\bf z})\cdot ({\bf x-y})$$ where ${\bf z}=\theta{\bf y}+(1-\theta){\bf x}$

Answer 3

There's a way to make this a little better. First of all, note that if we define $\alpha: \mathbb{R} \to \mathbb{R}^n$ by $\alpha(\theta)=\theta y+(1-\theta)x$ then $\alpha$ is a differentiable function of $\theta$ at every point of $\mathbb{R}$. Now, it's pretty clear that $g = f\circ \alpha$, because we have $g(\theta)=f(\alpha(\theta))$ which is what you want.

Now we simply apply the chain rule in it's general form (if you never saw this way of writing the chain rule, look at Spivak's Calculus on Manifolds):

$$Dg(\theta)=Df(\alpha(\theta))\circ D\alpha(\theta)$$

Now, in terms of matrices we have the following:

$$g'(\theta)=f'(\alpha(\theta))\alpha'(\theta)$$

However we have $\alpha'(\theta)=y-x$ so that understading $y-x$ as it's corresponding column matrix we have:

$$g'(\theta)=f'(\alpha(\theta))(y-x)$$

Now, since $f$ maps $\mathbb{R}^n \to \mathbb{R}$, we have $f'(\alpha(\theta))$ a row matrix and the expression above is the multiplication of the row matrix by the column matrix. Thinking of linear transformations, we are applying the transformation $Df(\alpha(\theta))$ into the vector $y-x$ so that we have:

$$g'(\theta) = Df(\alpha(\theta))(y-x)$$

Finally, if you want to write using gradients, remember that if $f$ is a scalar fields, applying it's derivative into some vector equals making the inner product between the gradient and the vector, so tht we obtain your expression:

$$g'(\theta) = \nabla f(\alpha(\theta))\cdot(y-x)$$

And just to finish a little geometrical interpretation: $\alpha$ is a curve, indeed $\alpha$ is a line. When you compute $f\circ \alpha$ you get the values of $f$ just along $\alpha$ (you restric $f$) and then $g'(\theta)$ is the derivative of the function along the line, so you are getting the directional derivative of $f$ in the direction of the line.