Given 2 vectors $\vec{s^{\prime}}$, $\vec{t^{\prime}}$ and $S^{\prime}=\vec{s^{\prime}} \cdot \vec{\sigma}$, $T=\vec{t^{\prime}} \cdot \vec{\sigma}$. The following equations are given
\begin{equation} \begin{aligned} \overrightarrow{s^{\prime}} \cdot \overrightarrow{t^{\prime}} &=\frac{1}{2} \operatorname{tr}\left[S^{\prime} T^{\prime}\right] \\ &=\frac{1}{2} \operatorname{tr}\left[\left(U S U^{\dagger}\right)\left(U T U^{\dagger}\right)\right] \\ &=\frac{1}{2} \operatorname{tr}[S T] \\ &=\vec{s} \cdot \vec{t} \end{aligned} \end{equation} Taken from this website (https://qubit.guide/3.4-unitaries-as-rotations.html)
I don't understand the first line (how the scalar product of 2 vectors is related to the product of the relevant matrices). Is this a special property of spin rotation operator ? A more detailed derivation of the first line or links to the topic would be much appreciated!
That's just a calculation. Let $$S = s_1 \sigma_1 + s_2 \sigma_2 + s_3 \sigma_3 \,(= \vec{s} \cdot \vec{\sigma})$$ $$T = t_1 \sigma_1 + t_2 \sigma_2 + t_3 \sigma_3 \,(= \vec{t} \cdot \vec{\sigma})$$
Then
$$ ST = s_1 t_1 \sigma_1^2 + s_2 t_2 \sigma_2^2 + s_3 t_3 \sigma_3^2 + \text{mixed terms} $$ You should know that
$$ \sigma_i \sigma_j = \delta_{ij}\, \text{Id} + \varepsilon_{ijk}\sigma_k$$ so the mixed terms are of the form $s_i t_j \sigma_k$ (times $\pm 1$). These have all zero trace (by inspection).
The terms with the squares are just of the form $s_i t_i \, \text{Id}$ where Id is the $2\times 2 $ identity matrix, which has trace $=2s_i t_i$, so their sum is twice the scalar product in question.
Note: the site you are looking at seems to assume that readers are willing to sit down with a piece of paper and derive some of the equations by themselves, e.g. by calculation. I personally think that this is a good thing, but it may be too fast a pace for others. If that applies to you consider using a textbook providing these details (I found, e.g., W. Scherer's book "Mathematics of Quantum Computing" quite comprehensive in that respect).