Let $$ \mathbf{H}=\sigma(\mathbf{Z}) $$ where $\sigma$ is an element-wise non-linear function, $\mathbf{H}\in\mathbb{R}^{m\times n}$, and $\mathbf{Z}\in\mathbb{R}^{m\times n}$. What is the index notation representation of $$\frac{\partial \mathbf{H}}{\partial \mathbf{Z}}$$
My attempt: $$\frac{\partial H_{ij}}{\partial Z_{pq}}=\delta_{ip}\delta_{jq}\sigma'(Z_{ij})$$
$ \def\s{\sigma}\def\t{\s^\prime} \def\p{\partial} \def\M{{\cal D}} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\qiq{\quad\implies\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} $Given a scalar function $\s(x)$ and its derivative $\t(x)$, we can express the differential of the elementwise application of the function to a matrix argument $Z$ by using an elementwise/Hadamard $(\odot)$ product $$\eqalign{ H &= \s(Z),\;\; K = \t(Z) \qiq dH &= K\odot dZ \\ }$$ To convert a Hadamard expression into index notation we need to extend the Kronecker delta from two indices to three, i.e. $$\eqalign{ {\vec\delta}_{jpr} &= \begin{cases} 1\quad{\rm if}\;\;j=p=r \\ 0\quad{\rm otherwise} \\ \end{cases} \\ }$$ Like the ordinary delta symbol, permuting the indices does not affect the value.
Using one extended delta symbol, we can multiply two vectors (using the Einstein convention) $$\eqalign{ c &= a\odot b \qiq c_p = a_i\,{\vec\delta}_{ipk}\,b_k \\ }$$ But to multiply matrices we need two such symbols $$\eqalign{ C &= A\odot B \\ C_{pq} &= A_{ij}\:{\vec\delta}_{ipk}{\vec\delta}_{jql}\:B_{kl} \;=\; A_{ij}\:\M_{ijpqkl}\:B_{kl} \\ \M_{ijpqkl} &= \begin{cases} 1 \quad {\rm if}\;i=p=k\;\;{\rm and}\;\;j=q=l \\ 0 \quad {\rm otherwise} \\ \end{cases} \\ }$$ Applying this to the differential expression above yields $$\eqalign{ dH_{pq} &= K_{ij}\,\M_{ijpqkl}\:dZ_{kl} \qiq \grad{H_{pq}}{Z_{kl}} = K_{ij}\,\M_{ijpqkl} \\ }$$ If you suspend the summation convention, then the result can be written as you have done in your post.