Derivative of a vector x wrt itself

Question

I am attempting to better understand matrix calculus. Given a vector $x$, I understand that: $\frac{\mathrm{d} x^t}{\mathrm{d} x}= I$ but struggle to think about $\frac{\mathrm{d} x}{\mathrm{d} x}$. This website says it is also the identity matrix, but I struggle to understand why. Would taking the derivative of a column vector wrt a column vector produce a tensor? Any intuition is greatly appreciated!

Answer 1

If you use the definition for vector-on-vector differentiation used on Wikipedia here, then the derivative of a column vector with respect to another column vector is indeed a matrix. To provide some intuition, consider $x'=f(x)$ as defining a coordinate transformation. (That is, $x_1'=f_1(x), x_2'=f_2(x)$, etc.) Then $dx'/dx$ is just the Jacobian matrix for this transformation, and it shouldn't be surprising that the identity transformation $f(x)=x$ has the identity matrix as its Jacobian.

Another route is to think in terms of linearization: If $y=f(x)$, then $$f(x+\epsilon h)=f(x)+ϵf′(x)h+\mathcal{O}(\epsilon^2)=y+ϵ (dy/dx) h+\mathcal{O}(\epsilon^2)$$ So the linearization of $f(x)$ only works if $(dy/dx)h$ is another column vector, i.e., $dy/dx$ is a matrix. (If you want a headache, try thinking about the $\epsilon^2$ terms!)