I apologise in advance for my vague and borderline senseless questions. I'm just trying to get a good hold on these things and I was having a tough time putting my thoughts into words.
I have two questions.
- $\nabla f (\vec{v})$ shows the direction of steepest ascent of f at v. I feel like the jacobian is a generalisation of gradient, for vector valued functions. So what meaning does the jacobian matrix have when evaluated at a point in the input space?
- $[\nabla f \cdot \vec{u}]({\vec{v}}) = \nabla_{\vec{u}} f(\vec{v})$ gives the directional derivative of f along u, which captures how f changes at v as one moves along u. Can I say that (Jac f)$u$ evaluated at v captures how f changes at v as one moves along u in the input space? Further comments and elaborations on this are greatly appreciated.
SETUP, stuff I figured out:
Consider $$f: R^m \rightarrow R^n.$$ $$ f: \vec{x} = \begin{pmatrix} x_1 \\\ x_2 \\\ ... \\\ x_m\end{pmatrix} \rightarrow f(\vec{x}) = \begin{pmatrix}f_1(\vec{x}) \\\ f_2(\vec{x}) \\\ ... \\\ f_n(\vec{x})\end{pmatrix} $$
Then the jacobian $$\text{Jac f}(\vec{x}) = \begin{pmatrix} \nabla f_1(\vec{x}) \\\ \nabla f_2(\vec{x}) \\\ ... \\\ \nabla f_n(\vec{x}) \end{pmatrix}$$ [This is a nice equivalent way of writing the jacobian]
Note that Jac f $: R^m \rightarrow R^n$ as well.
Now I want to examine how the jacobian transforms any input vector (from $R^m$).
By standard matrix multiplication, $$ \text{(Jac f) }\vec{v} = \begin{pmatrix} \nabla f_1 \cdot \vec{v} \\\ \nabla f_2 \cdot \vec{v} \\\ ... \\\ \nabla f_n \cdot \vec{v} \end{pmatrix} = \begin{pmatrix} \nabla_{\vec{v}}f_1 \\\ \nabla_{\vec{v}}f_2 \\\ ... \\\ \nabla_{\vec{v}}f_n\end{pmatrix} $$
[The notation $\nabla_{\vec{v}}f$ denotes the directional derivative of f along the vector v.]
Thus we note that the Jacobian applied on a vector v collects the directional derivatives of each component function of f along the vector v.
Everything you said makes sense. The only thing I would add is the following. In single-variable calculus, the derivative represents the "slope" of the linear approximation. That is, near the point $x=p$ the graph looks like $f(x) \approx f(p) + f'(p)(x-p)$.
The idea in higher dimensions is the same. The Jacobian encodes the coefficients ("slopes") of the linear approximation of $f \colon \Bbb{R}^m \to \Bbb{R}^n$. In other words, near $\vec{v} \in \Bbb{R}^m$, the function $f$ is approximated by $$f(\vec{x}) \approx f(\vec{v}) + (\mathrm{Jac}(f(\vec{v})) \cdot (\vec{x}-\vec{v})$$
If you move the $f(\vec{v})$ to the other side, and substitute $\vec{u}=\vec{x}-\vec{v}$, you have $f(\vec{v}+\vec{u})-f(\vec{v}) \approx (\mathrm{Jac}(f(\vec{v})) \cdot \vec{u}$. This was what you asked/claimed in question #2: that $(\mathrm{Jac}(f(\vec{v}))\vec{u}$ captures how the value of $f$ changes under small changes in the input in the direction $\vec{u}$.