I have a question about an equality on page 16 in Alvarez's famous work Axioms and fundamental equations of image processing. I will try to make my question self-contained. Please remind me of anything unclear.
Domain and definition:
$x\in \mathbb{R}^N$, function $f:\mathbb{R}^N \rightarrow \mathbb{R}$, $T_{t}$ is an linear operator on $f$, such that $T_tf: \mathbb{R}^N \rightarrow \mathbb{R}$ is also a function on $\mathbb{R}^N$
The author defines [Isometry invariance] as below
$T_{t}(R \cdot f)=R \cdot T_{t}(f)$ for all $f, t \geqq 0$ and for all transforms $R$ defined by $(R \cdot f)(x)=f(R x)$ where $R$ is an orthogonal transform of $\mathbb{R}^{N}$.
Suppose $\frac{T_tf-f}{t}$ admits a limit when $t\rightarrow 0$, and the limit can satisfy a second-order PDE, i.e. $$\lim_{t\rightarrow 0}\frac{T_tf-f}{t}=F(H,p)$$
where $F$ is some function on $\mathbb{R}^{N\times N}\times \mathbb{R}^{N}$. $H=\nabla^2 f, p=\nabla f$.
Problem: The author states that, if we have [Isometry invariance], then $$F(R^T H R, R^T p)= F(H,p)$$ for any orthogonal matrix R.
My attempt: I omit the limit notation $\lim_{t\rightarrow 0}$ for simplicity \begin{align} F(H,p)&= \frac{T_tf-f}{t} \\ &= \frac{T_t(R^T\cdot R \cdot f)-R^T\cdot R \cdot f}{t} \qquad \text{(Orthogonal)} \\ &= \frac{R^T T_t(R \cdot f)-R^T \cdot R \cdot f}{t} \qquad \text{(Isometry invariance)} \\ &= R^T \cdot \frac{T_t(Rf)-Rf}{t} \\ &= R^T F(\nabla^2 (Rf) ,\nabla (Rf)) \qquad \text{(Definition)}\\ &= F(R^T H R, R^T p) \qquad \text{(Last equality, WHY?)}\\ \end{align}
I got stuck here. Specifically, in the second term, I don't know why $$R^T \nabla (Rf) = R^T p=R^T \nabla f$$
Any suggestions are welcomed.
I've worked out this question. The key mistake is that the derivative should be w.r.t $R^Tx$ rather than $x$.
To avoid misunderstanding, I would like to denote $\hat{R}$ as an operator, and $R$ as a matrix. The definition of operator $\hat{R}$ holds: $$\hat{R}f(x)=f(Rx)$$ Then we have \begin{align} & \quad ~ F(\nabla^2 f(x) ,\nabla f(x))\\ & = F(\nabla^2 f(RR^Tx) ,\nabla f(RR^Tx)) \\ & = \lim_{t\rightarrow 0} \frac{T_t f (RR^Tx)-f(RR^Tx)}{t}\\ & = \lim_{t\rightarrow 0} \frac{\hat{R}T_t f (R^Tx)-\hat{R}f(R^Tx)}{t} \qquad \text{(Definition)} \\ & = \lim_{t\rightarrow 0} \frac{T_t \hat{R}f (R^Tx)-\hat{R}f(R^Tx)}{t} \qquad \text{(Isometry)} \\ & = F(\nabla_{R^Tx}^2 \hat{R}f(R^Tx) ,\nabla_{R^Tx} \hat{R}f(R^Tx)) \end{align} Denote $R^Tx=y$, then $$\nabla_{R^Tx} \hat{R}f(R^Tx)= \frac{\partial \hat{R}f(y)}{\partial y}=\frac{\partial f(Ry)}{\partial y}=R^T \nabla f(Ry)= R^T \nabla f(x)$$ $$\nabla^2_{R^Tx} \hat{R}f(R^Tx)= \frac{\partial R^T \nabla f(Ry)}{\partial y}=R^T \nabla^2 f(Ry) R = R^T \nabla^2 f(x) R$$ q.e.d.