$\newcommand{\psym}{\operatorname{P}_{\ge 0}}$ $\newcommand{\Sig }{\Sigma}$
Let $\psym$ denote the subset of symmetric positive semi-definite matrices.
Let $S:\psym \setminus \{0\} \to \psym \setminus \{0\}$,
where $S(A)=\sqrt{A}$ is the unique positive semi-definite square root of $A$.
$\psym \setminus \{0\}$ is a manifold with boundary. I am trying to prove $S$ is not differentiable at every point in $\{A \in \psym | \, \, \det(A)=0 \}$ (i.e, on the boundary).
Am I correct about this claim? and its proof?
Here is my attempt:
Assume it was differentiable at such an $A$. Since $S^{-1}(A)=A^2$ is differentiable everywhere, we would get:
$$ Id=d(S^{-1} \circ S)_A = dS^{-1}_\sqrt{A} \circ dS_A \Rightarrow dS^{-1}_\sqrt{A} \text{ is invertible}$$
But this is false since $dS^{-1}_A(X)=AX+XA$ is non-invertible. Let's see why.
First, note that $dS^{-1}_A:\operatorname{sym}_n \to \operatorname{sym}_n$ (where $\operatorname{sym}_n$ is the space of symmetric matrices).
We want to show there is a symmetric, non-zero $X$ such that $AX+XA=0$. Since $A$ is symmetric we can orthogonally diagonalize it: $A=V \Sig V^T, V \in O_n$
Then $AX+XA=V \Sig V^T X+XV \Sig V^T=0 \iff \Sig (V^TXV) + (V^TXV) \Sig =0$
Since $X$ is non-zero and symmetric $\iff$ $V^TXV$ is non-zero and symmetric,
it is enough to show this for non-zero diagonal matrices* $\Sig$ such that $\det(\Sig)=0$.
In this case, the equation becomes $X_{ij}(\sigma_i+\sigma_j)=0$. Assume without loss of generality $\sigma_1 = 0$ (since $\det(\Sig)=0$), and choose $X_{11}=1$ and all the other $X_{ij}$ to be zero.
Is this proof true? Is there an easier argument?
*There are other, perhaps easier ways to see it is enough to consider the diagonal case only. (For example, using the fact that taking squares commutes with conjugation, and orthogonal conjugation is a self-diffeomorphism)
Your reasoning is correct; yet, you can do the job faster as follows: let $u\in\ker(A)$ be a (non-zero) eigenvector. Put $X=uu^T$ (a symmetric matrix). Thus $AX+XA=(Au)u^T+u(Au)^T=0$ and we are done.
EDIT 1. To @ Asaf Shachar . Let $M=P_{\geq 0}$. Then $M$ is a closed cone; in particular, it is a convex set. Yet, the edge $E$ can be complicated; if $n=2$, then $E$ has a sole singular point, the $0$ matrix. When $n=3$, the matrices $A_0=\begin{pmatrix}a&b&c\\b&d&e\\c&e&f\end{pmatrix}$ s.t. $df=e^2,c^2=af,b^2=ad$ are singular points of $E$. Clearly, $E$ admits no tangent space in these points.
Finally we must define the derivative only for the symmetric matrices $X$ s.t. $A_0+X\in M$ (obviously, this set is no more a vector space). My example above is convenient because $X=uu^T\geq 0$ and consequently, $A_0+X\in M$.
EDIT 2. Assume $n=3$. Then $M$ is defined by $a\geq 0,ad-b^2\geq 0,\det(A)\geq 0$. Consider $A_0\in E$ s.t. $a>0,ad-b^2>0$. Then, in a neighborhood of $A_0$, $M$ is defined by $\{A\in Sym_n|f(A)=\det(A)\geq 0\}$; $f^{-1}(0)$ is locally a manifold (of dimension $n(n+1)/2-1$) when the derivative $Df(A_0)$ is non-zero. Assume that $Df(A_0)=0$ and note that $Df(A_0):H\in Sym_n^+\rightarrow tr(adj(A_0)H)$; since $A_0\geq 0$, $adj(A_0)$ too and $Df(A_0)(adj(A_0))=0\implies adj(A_0)=0$, that is $rank(A_0)\leq n-2= 1$. That is contradictory because $ad-b^2>0$.
Thus, if $E$ admits a singular point in $A_0$, then $a=0$ or $ad-b^2=0$; if $a=0$, then $b=0$ and $ad-b^2=0$. In the same way, we can prove that necessarily, one has also $df-e^2=0,af-c^2=0$.
Conversely, assume that $df=e^2,c^2=af,b^2=ad$; since $A_0\geq 0$, $\det(A_0)=0$ and moreover $rank(A_0)\leq 1$. Finally, up to a change of orthonormal basis (and an homothety), we may assume that $A_0=0_3$ or $A_0=diag(1,0,0)$. Let $A_0=diag(1,0,0)$ and $X=\begin{pmatrix}u&v&w\\v&x&y\\w&y&z\end{pmatrix}$. Locally, $E$ is
$\{A_0+X|(1+u)x-v^2=0\;and\;\det(A_0+X)> 0\}\cup$
$\{A_0+X|(1+u)x-v^2> 0\;and\;\det(A_0+X)= 0\}\cup$
$ \{A_0+X|(1+u)x-v^2= 0\;and\;\det(A_0+X)= 0\}=S_1\cup S_2\cup S_3$.
Let $X$ be a small matrix s.t. $\det(A_0+X)=0,(1+u)x-v^2=0,wx\not= 0$, that is equivalent to $u+1=v^2/x,vy-wx=0,wx\not= 0$. Note that $A_0+X\in S_3$. The two half hypersurfaces $\overline{S_1},\overline{S_2}$ transversally intersect in $A_0+X$ because their normal vectors in $A_0+X$ are not parallel. Finally $E$ is not a $C^1$ manifold.
EDIT 3. Answer to @ Asaf Shachar . You are right, I gave only necessary conditions. Yet, implicitly, I use all the conditions about the principal minors when I write " In the same way, we can prove that necessarily, one has also $df-e^2=0,af-c^2=0$." Thus I should rewrite certain details of the proof in EDIT 2., but the final result is correct: when $n=3$, $E$ has no tangent space in any point $A\in E$ s.t. $rank(A)\leq 1$. More generally, for every $n$, $E$ has no tangent space in any point $A\in E$ s.t. $rank(A)\leq n-2$. In particular, $E$ is a $C^0$ but not $C^1$ algebraic set.