Suppose $f:\mathbb{R}^p \to \mathbb{R}$ is a smooth function and we define a function $g: x \mapsto f[x(x^Tx)^{-1/2}].$ Thus, $g$ depends only on the direction of its argument, not the magnitude. The function $g$ is smooth on the set $\mathbb{R}^p\setminus{0}$ but is not defined at $0$ and $\lim_{x\to 0} g(x)$ depends upon which direction you approach from.
We can define a class of functions $g_{\epsilon}: x \mapsto f[x(x^Tx + \epsilon)^{-1/2}]$ which are better behaved in the sense that for any $\epsilon >0,$ the function $g_\epsilon$ is smooth everywhere on $\mathbb{R}^p.$ Then $g$ can be recovered as $\lim_{\epsilon\to 0} g_{\epsilon}(x) = g(x)$ for every $x\in \mathbb{R}^p\setminus\{0\}.$
I'm interested in knowing whether $$\lim_{\epsilon\to0} \frac{\partial}{\partial x_{i}} g_\epsilon(x) = \frac{\partial}{\partial x_{i}} g(x)$$ everywhere that the right hand side is defined. In other words, can we exchange the order of partial differentiation with respect to an element of the vector $x$ and taking the limit with respect to $\epsilon?$ Is this exchange possible for arbitrary mixed partial derivatives, i.e. do we also have $$\lim_{\epsilon\to0} \frac{\partial}{\partial x_{i_1}} ... \frac{\partial}{\partial x_{i_m}} g_\epsilon(x) = \frac{\partial}{\partial x_{i_1}} ... \frac{\partial}{\partial x_{i_m}} g(x)?$$
I'm aware that, in general, exchanging limits/differentiation requires strong assumptions such as uniform convergence of the sequence of derivatives. However, I'm hoping it's true for the construction I've described. It's not a problem if we need to assume more about $f$ (within reason), e.g. $f$ and/or its derivatives are bounded on the unit sphere.
More generally, what if $f, g,$ and the $g_\epsilon$ take a matrix argument $X \in \mathbb{R}^{p \times k}$ but are otherwise defined analogously? To be more precise, $f: \mathbb{R}^{p\times k} \to \mathbb{R}$ is a smooth function and $$g: X \mapsto f[X(X^TX)^{-1/2}]$$ $$g_{\epsilon}: X \mapsto f[X(X^TX + \epsilon I )^{-1/2}].$$ In this case, the function $g$ only depends on $X$ through the (rectangular) orthogonal matrix $X(X^TX)^{-1/2}$ and is smooth on the set of full rank matrices. The function $g_{\epsilon}$ is smooth everywhere on $\mathbb{R}^{p \times k}$ and $\lim_{\epsilon \to 0} g_{\epsilon}(X) = g(X)$ for all full rank $X.$
Can we exchange the order of taking arbitrary mixed partial derivatives of $g_\epsilon$ with respect to the entries of $X$ and the limit with respect to $\epsilon?$ Again, it's no problem to require more of $f$ (within reason), e.g. that $f$ and/or its derivatives are bounded on the set of orthogonal matrices.