Let $P$ be an $n\times n $ positive semidefinite matrix over $\mathbb{C}$, let $p\in\mathbb{R}$ be in the range $0<p<1$. Consider the function $g:[0,\infty)\rightarrow\mathbb{R}$ defined by $g(x)=x^p$ (which may be extended to positive matrices) and consider the function defined by $$ f(A) = \operatorname{Tr}(P\,g(A)) = \operatorname{Tr}(PA^p) $$ for all other positive semidefinite $n\times n$ matrices $A$. This function is differentiable at all positive definite matrices $A$. Indeed, the function $g(X)=X^p$ is diferentiable at $A$ if $A$ is positive definite. See for example Theorem 3.23 and 3.25 here (or section X.4 of Bhatia's book on Matrix Analysis).
However, I would like to determine the directional derivatives of $f$ at matrices $A$ that are not positive definite.
Suppose $A$ is a positive $n\times n$ matrix satisfying $\operatorname{im}(P)\subseteq\operatorname{im}(A)$. Let $B$ be another $n\times n$ hermitian matrix such that the matrix $A+tB$ is positive semidefinite for all $t>0$ small enough.
My main question is: How can we determine when the following directional derivative exists, and how can we compute it? $$ df(A;B) = \lim_{t\rightarrow 0} \frac{f(A+tB)-f(A)}{t} = \lim_{t\rightarrow 0} \frac{\operatorname{Tr}(P(A+tB)^p)-\operatorname{Tr}(PA^p)}{t} $$
Intuitively, it seems like we should be able to compute it as follows.
Suppose without loss of generality that $A$ is diagonal with $A=\operatorname{diag}(a_1,\dots,a_n)$ with $r$ positive eigenvalues $a_1,\dots,a_r>0$ and $n-r$ zero eigenvalues $a_{r+1}=\cdots=a_n=0$. Define the $n\times n$ matrix $D$ whose entries are $$ D_{i,j} = \left\{\begin{array}{ll} \frac{a_i^p-a_j^p}{a_a-a_j}&\text{if }a_i\neq a_j\text{ and }a_i,a_j\neq0\\ p a_i^{p-1} & \text{if }a_i=a_j\neq 0\\ 0 & \text{if }a_i=0\text{ or }a_j=0 \end{array} \right. $$ and consider the entrywise product of $D\circ B$ whose entries are $(D\circ B)_{ij} = D_{ij}B_{ij}$. If $A$ is positive definite, the desired directional derivative is $$ df(A;B) = \operatorname{Tr}(P (D\circ B)). $$ I think the same should also hold in the case when $A$ is not positive definite. In the case when $A$ is positive but not positive definite, the matrix $D$ defined above is just zero outside of the image of $A$. And since we assume $\operatorname{im}(P)\subseteq\operatorname{im}(A)$ we can just throw everything out that is not in this subspace. This seems to work, but I can't find a rigorous argument as to why.
How might I prove this fact?
Further, what can we say about directional derivatives in the case when $p<0$? Or more generally, consider an arbitrary function $g:(0,\infty)\rightarrow\mathbb{R}$ and define the function $f$ on positive matrices as $$ f(A)=\operatorname{Tr}(P\, g(A)). $$ For example, consider $f(A)=\operatorname{Tr}(P\log(A))$, which is well defined for even for semidefinite singular operators $A$ as long as $\operatorname{im}(P)\subseteq\operatorname{im}(A)$. What can we say about the directional derivatives $df(A;X)$ in this case?
I think that it is better to consider an arbitrary hermitian $P$ ($P$ does not need to be $\geq 0$) and a particular $B$.
We put $g(0)=0$. Let $A=diag(\Delta_{n-r},0_{r})$ where $\Delta >0$ is diagonal. If $U$ is some matrix, then $D_U$ denotes the matrix $D$ associated to $U$ along the OP's definitions. We assume that $B=diag(Q_{n-r},T_r)$ where $T\geq 0$ and $Q$ hermitian; note that $A+tB\geq 0$ for $t>0$ small enough. Let $P=\begin{pmatrix}U_{n-r}&V\\V^*&X_r\end{pmatrix}$.
Note that $D_A=diag(D_{\Delta},0)$, $D_A\circ B=diag(D_{\Delta}\circ Q,0)$, $tr(PA^p)=tr(U\Delta^p)$ and $(**)$ $tr(P(D_A\circ B))=tr(U(D_{\Delta}\circ Q))$.
$(*)$ We know that $(tr(U\Delta ^p))'(Q)=tr(U(D_{\Delta}\circ Q))$.
$\dfrac{1}{t}tr(P(A+tB)^p-PA^p)=$
$\dfrac{1}{t}tr(\begin{pmatrix}U&V\\V^*&X\end{pmatrix}diag((\Delta+tQ)^p,t^pT^p)-\begin{pmatrix}U\Delta^p&0\\V^*\Delta^p&0\end{pmatrix})=$
$\dfrac{1}{t}tr(U(\Delta+tQ)^p-U\Delta^p)+\dfrac{t^p}{t}tr(XT^p)$.
When $t\rightarrow 0_+$, the first term of the RHS tends to $tr(P(D_A\circ B))$ (cf. $(*)$ and $(**)$). The second term has a limit iff $tr(XT^p)=0$. If you want that $P\geq 0$, then $X\geq 0$ and, since $T^p\geq 0$, necessarily $XT^p=0$.
$\textbf{Proposition}$. If $B$ satisfies $B=diag(Q,T)$ where $T\geq 0$ and $tr(XT^p)=0$, then $f'_{t>0}(A)(B)=tr(P(D_A\circ B))$.