When is a Jacobian Matrix not diagonal?

Question

According to this article (See: the section "Derivatives of vector element-wise binary operators"), Jacobian Matrix is nothing but a Diagonal Identity Matrix.

I am failing to understand What is so special about Jacobian Matrix then? Why don't we talk about an Identity matrix then?

When is a Jacobian Matrix not diagonal?

Answer 1

From the article,

That's quite a furball, but fortunately the Jacobian is very often a diagonal matrix, a matrix that is zero everywhere but the diagonal.

is just an empirical claim that most of the Jacobians that you meet in this domain (deep learning according to this article) are diagonal. It is not true in general.

Answer 2

The given answer states the argument as a matter of fact merely referencing the original source. We can and should do better explaining why the stated fact holds. Below is an attempted answer of substance without a claim that it is either optimum or without error:

'WHAT HAPPENS IN VEGAS STAYS IN VEGAS'

Jacobian matrix is an $n \times m$ matrix of a vector-valued function's first-order partial derivatives i.e. gradients of this function's components with respect to each argument vector's component.

A Jacobian is diagonal when it contains gradients of an element-wise vector-valued function $f(x)$ such that: $$f: \Bbb{R}^n \to \Bbb{R}^m: [x_1, x_2, ..., x_n] \to [f_1, f_2, ..., f_m],$$ $$f(x): \Bbb{R}^n \to \Bbb{R}^m, n = m$$ i.e. the function does not change dimensionality from input to output

A Jacobian matrix is diagonal if it characterizes an element-wise operation. This is because element-wise operations do not change output vector dimensionality (are dimension-preserving) i.e. they exclude vector multiplication - whether it is a dot product or a matrix multiplication i.e. the operations that may change output dimension). In case of dot product, the output vector dimensionality is collapsed to zero vector $(0_1,0_2, .. 0_n)$. In case of matmul, $n ≠ m$ i.e. $n > m$ or $n < m$ (output dimensions are reduced or expanded).

Jacobians of element-wise vector-valued functions are diagonal because element-wise operations are 'dimension-wise' or dimension-preserving i.e. they constrain effect of argument components changes to the corresponding function's components (dimension). In essence, 'What happens in Vegas stays in Vegas' captures this dimension-preserving property of element-wise operations, which render all but diagonal elements of the Jacobian zero.

P.S. Dimension-preserving operations are not necessarily linear. Vector functions (operations) can be element-wise and non-linear e.g. Hadamard product (element-wise vector multiplication).