Suppose I have a function $\Lambda: S^\text{tr} \to S^\text{tr}$ where $S^\text{tr} = \{Q \in \mathbb{R}^{n \times n} \: |\: Q = Q^T \: \text{and} \: \text{tr}(Q) = 0\}$ is the space of traceless, symmetric matrices. I would like to take a Gateaux derivative in the direction of $\delta Q$: \begin{equation} d\Lambda(Q;\delta Q) = \left.\frac{d}{d\tau} \Lambda(Q + \tau \, \delta Q)\right|_{\tau = 0} \end{equation} It would be useful to write down $d\Lambda(Q; \delta Q)$ in index notation. For an example of what I mean, if it were that $\Lambda: \mathbb{R}^{n\times n} \to \mathbb{R}^{n \times n}$, we may write this explicitly as: \begin{equation} \begin{split} \left.\frac{d}{d\tau} \Lambda_{ij} \left(Q_{kl} + \tau \, \delta Q_{kl}\right) \right|_{\tau = 0} &= \frac{d}{d \tau} \left[\Lambda_{ij} \left(Q\right) + \tau \left.\frac{\partial \Lambda_{ij}}{\partial Q_{kl}}\right|_Q \delta Q_{kl} + \mathcal{O}\left(\tau^2\right) \right] \\ &= \left.\frac{\partial \Lambda_{ij}}{\partial Q_{kl}}\right|_Q \delta Q_{kl} \end{split} \end{equation} where here $\Lambda$ is assumed smooth around $Q$. My issue is that, when considered explicitly in the subspace $S^\text{tr}$, this appears to be nonsensical. That is, for $n = 3$ we may identify elements of $S^\text{tr}$ with elements of $\mathbb{R}^5$ as: \begin{equation} Q = \begin{bmatrix} Q_1 &Q_2 &Q_3 \\ Q_2 &Q_4 &Q_5 \\ Q_3 &Q_5 &-(Q_1 + Q_4) \end{bmatrix} \end{equation} with $Q_i$ entries in an element in $\mathbb{R}^5$. Then we may almost calculate $\partial \Lambda_{ij} / \partial Q_{kl}$ with this mapping, except that $\partial \Lambda_{ij} / \partial Q_{33}$ does not make sense as (to my knowledge) there's not a reasonable notion of differentiating by a sum.
Alternatively, is it valid to understand $\Lambda: \mathbb{R}^5 \to S^\text{tr}$ and notate as: \begin{equation} d\Lambda(Q; \delta Q) = \frac{\partial \Lambda_{ij}}{\partial Q_k} \delta Q_k \end{equation} with $k$ indexing entries in $\mathbb{R}^5$? The purpose of taking the Gateaux derivative is to set up a Newton-Rhapson iteration scheme, for context.