Suppose that $\mathbf{Z}$ is a design matrix of a linear regression.
Where $\mathbf{Z}^{T}\mathbf{Z}$ is a symmetric matrix.
Also suppose that $\mathbf{H}$ is the hat matrix defined as follows:
$\mathbf{H} = \mathbf{Z} \hat{\boldsymbol \beta} = \mathbf{Z} \left( \mathbf{Z}^\textsf{T} \mathbf{Z} \right)^{-1} \mathbf{Z}^\textsf{T}$
According to my textbook we can now we can say that
$trace(\mathbf{H}) = trace(\mathbf{Z}(\mathbf{Z}^{T}\mathbf{Z})^{-1}\mathbf{Z}^{T}) = tr[(\mathbf{Z}^{T}\mathbf{Z})^{-1}(\mathbf{Z}^{T}\mathbf{Z})]$
And the above is due to the following property of traces namely:
$\mathbf{x}'\mathbf{A} \mathbf{x} = trace(\mathbf{A}\:\mathbf{x}\:\mathbf{x}')$
I'm not sure how this property of traces was applied here. Can someone provide clarity in which context the property was used to attain the above-mentioned equality.
Hint: For $A$ (format $n\times p$) and $B$ (format $p\times n$) we have $\text{tr}(AB)=\text{tr}(BA)$. In your case $A=Z$ and the rest is $B$.
The property below is similar as $\mathbf{x}^T\mathbf{Ax}$ is a scalar and the trace of a scalar is the scalar itself. Hence, we have
$$\mathbf{x}^T\mathbf{Ax} = \text{tr}\left[\mathbf{x}^T\mathbf{Ax}\right]$$
Now, you can apply the same rule.