Why does $|tr(XY)| \leq tr(|XY|)$ hold for any complex matrices where $|XY|$ denotes $\sqrt{Y^*X^*XY}$?
would following proof be correct? So the trace of a matrix $A$ is the sum of its eigenvalues and the the trace of $|A|$ is the sum of the singular values: $$|tr(A)|=|\sum \lambda_j|\leq \sum s_j=tr(|A|) $$
I do not understand why you consider $XY$ since $X$ and $Y$ always occur in the combination $XY$. Thus setting $Z=XY$ you are asking why $$|\operatorname{tr}Z|\leqslant\operatorname{tr}|Z|\,.$$ It often happens that one can write $$Z=U|Z|,\;|Z|=\sqrt{Z^\ast\!Z}$$ where $U$ is a partial isometry. This is the case for closed densely defined operators in Hilbert space, so applicable to your case
(https://en.wikipedia.org/wiki/Polar_decomposition, T.Kato: "Perturbation theory of linear operators", see polar decomposition of a closed operator)
Then $$|\operatorname{tr}Z|=|\operatorname{tr}(U|Z|)|\leqslant \|U\|\, \operatorname{tr}|Z|\leqslant\operatorname{tr}|Z|$$ where $\|U\|$ is the operator norm of $U$. $U$ being a partial isometry, $\|U\|\leqslant 1$.
Example $$Z = \begin{pmatrix}1 & -1 \\0 & 0\end{pmatrix} ,\;Z^\ast Z=\begin{pmatrix}1 & 0 \\-1 & 0\end{pmatrix} \begin{pmatrix}1 & -1 \\0 & 0\end{pmatrix} = \begin{pmatrix}1 & -1 \\-1 & 0\end{pmatrix} \\[3ex] \begin{pmatrix}1 & -1 \\0 & 0\end{pmatrix} = \begin{pmatrix}1 & 0 \\0 & 0\end{pmatrix} \begin{pmatrix}1 & -1 \\-1 & 0\end{pmatrix} ,\;U=\begin{pmatrix}1 & 0 \\0 & 0\end{pmatrix}$$