Let $U:\text{Hom}(\mathbb{R}^d,\mathbb{R}^d) \to \mathbb{R}$ be a smooth function .
If $U$ is orthogonally-invariant, i.e. $U(QA)=U(A)$ for every $Q \in \text{SO}(n),A \in \text{Hom}(\mathbb{R}^d,\mathbb{R}^d)$, then
$$ dU_{QA}(QB)=dU_{A}(B) \tag{1}$$
Does the converse hold? That is, suppose $U$ is a $C^{\infty}$ function whose differential satisfies equation $(1) $ for every $A,B\in \text{Hom}(\mathbb{R}^d,\mathbb{R}^d),Q\in \text{SO}(n)$. Is it true that $U(QA)=U(A)$ for every $Q \in \text{SO}(n)$?
The answer is positive. Indeed, define $h:\text{SO}(n) \to \mathbb{R}$, by $$ h(Q)=U(QA).$$
Then, $dh_Q(\tilde B)=dU_{QA}(\tilde BA)=dU_{A}(Q^{-1}\tilde BA)=dh_I(Q^{-1}\tilde B)$, so $$ dh_Q = dh_I \circ Q^{-1} \,\,\,\text{ for every } Q \in \text{SO}(n). \tag{1}$$
Since $\text{SO}(n)$ is compact, $h$ obtains a maximum at some point $P \in \text{SO}(n)$. Thus, $dh_P=0$, hence by equation $(1)$, $dh_I=0$, and so $dh_Q =0$ for every $Q \in \text{SO}(n)$. This implies $h$ is constant.
Comment: This argument use the compactness of the symmetry group $\text{SO}(n)$ in an essential way; indeed, for non-compact groups $G$, "differential invariance" of a function $G \to \mathbb{R}$ does not imply it is constant. Take e.g. $G=\mathbb{R}$, and $f(x)=x$.