Consider an arbitrary rotationally (and translationally) invariant smooth function,
\begin{align} f(x,y)=f(|x-y|^{2}),\end{align}
where $|x-y|^{2}\equiv(x-y)\cdot(x-y)$ with $x,y\in\mathbb{R}^{n}$ and "$\cdot$" denotes the usual Euclidean inner product.
Suppose one applies a rotation $R$ (about the origin) to the $x$-variable--i.e., $f(x,y)\to f(Rx,y)$. This operation should be equivalent to instead applying the inverse rotation $R^{-1}$ to the $y$-variable, \begin{align}f(Rx,y)=f(x,R^{-1}y)\label{1}\tag{1}.\end{align} This follows from the fact that for rotations, $|Rx|^{2}=|x|^{2}$ and $(R^{-1})=R^{T}$, thus, \begin{align} |Rx-y|^{2} &=|Rx|^{2}+2(Rx)\cdot y+|y|^{2} \\ &=|x|^{2}+2x\cdot (R^{T}y)+|R^Ty|^{2}\\ &=|x-R^{T}y|^{2}\\ &=|x-R^{-1}y|^{2}. \end{align}
Now, consider a "small" rotation $I+\delta R$ with $\delta R$ an infinitesimal matrix. To linear order in $\delta R$, the group property implies, \begin{align}\delta R^{T}=-\delta R, \end{align} and $(I+\delta R)^{-1}=I-\delta R$. For this small rotation,
\begin{align}f((I+\delta R)x,y) &\approx f(x,y)+\delta Rx\cdot\partial_{x}f(x,y)+\mathcal{O}(\delta R^{2})\\ &\approx f(|x-y|^{2})+\delta Rx\cdot(x-y)f'(|x-y|^{2})+\mathcal{O}(\delta R^{2}),\tag{2}\label{2} \end{align} using the chain rule in going to the second line. But according to \eqref{1}, this should be equal to, \begin{align} f(x,(I-\delta R)y) &\approx f(x,y)-\delta Ry\cdot\partial_{y}f(x,y)+\mathcal{O}(\delta R^{2})\\ &\approx f(|x-y|^{2})+\delta Ry\cdot(x-y)f'(|x-y|^{2})+\mathcal{O}(\delta R^{2}) \tag{3}\label{3} \end{align}
But it would seem that $\eqref{2}=\eqref{3}$ only if, $\delta Rx\cdot(x-y)=\delta Ry\cdot (x-y)$, which does not look obviously equivalent to me.
I've stared at this for a while, but cannot spot the error in my manipulations.
I believe equations (2) and (3) do, indeed, follow from the setup and the ("non-obvious") necessary condition for their equivalence,
\begin{align} \delta Rx\cdot(x−y)=\delta Ry\cdot(x−y), \label{*}\tag{$\star$} \end{align}
does, in fact, hold in general: Note the anti-symmetry required by the group condition, $\delta R^T=-\delta R$, implies $\delta Rx\cdot x=0=\delta Ry\cdot y$ and $\delta Rx\cdot y=-\delta Ry \cdot x$ which together imply \eqref{*} holds.