I'm an engineering student and I ran into something like this while studying mechanics of the continuum:
$\vec{\eta*} = [N]\cdot\vec{\eta}$ where $[N]$ is the rotational matrix and it represents the rotation which leads from the frame of reference x,y,z to the frame of reference l,m,n:
$[N]= \begin{vmatrix} lx & ly & lz\\ mx & my & mz\\ nx & ny & nz\\ \end{vmatrix} $
where lx, ly, ...nz are the direction cosines of the angles between the unit vectors l and x, l and y, ...n and z of the two frames of reference.
Proving that the rotational matrix is equivalent to the matrix of the direction cosines is straightforward in two dimensions. In fact, considering an anticlockwise rotation of the frame of reference from $x,y,z$ to $l,m,n$ by an angle $\theta$, with $z$ being the rotational axis (Image), the rotation matrix can be derived to be as follows:
$[N]= \begin{vmatrix} \cos\theta & \sin\theta & 0\\ -\sin\theta & \cos\theta & 0\\ 0 & 0 & 1\\ \end{vmatrix} = \begin{vmatrix} \cos\theta & \cos\bigl(\frac{\pi}{2}-\theta\bigr) & 0\\ \cos\bigl(\frac{\pi}{2}+\theta\bigr) & \cos\theta & 0\\ 0 & 0 & \cos0\\ \end{vmatrix} = \begin{vmatrix} lx & ly & 0\\ mx & my & 0\\ 0 & 0 & nz\\ \end{vmatrix} $
What I can't figure out is: how can I prove this holds true for every rotation in 3D space?
Let us identify, by a slight abuse of notation, the initial $b=(b_1,b_2,b_3)$ and final $B=(B_1,B_2,B_3)$ bases, assumed orthonormal ( = orthogonal + normalized) with their matrices:
$$b:=\begin{pmatrix}|&|&|\\b_1&b_2&b_3\\|&|&|\end{pmatrix} \ \text{and} \ B:=\begin{pmatrix}|&|&|\\B_1&B_2&B_3\\|&|&|\end{pmatrix}$$
(entries = coordinates of the different vectors with respect to the standard basis of $\mathbb{R}^3$).
The connection between direction cosines matrix $C$ and rotation matrix $R$ is captured into the following matricial expression as an equivalence up to a change of basis:
$$\require{AMScd}\begin{CD} \mathbb{R^n} @>{C}>> \mathbb{R^n} \\ @VV{\text{b}}V @AA{\text{b}^T}A \\ \mathbb{R^n} @>{R}>> \mathbb{R^n} \end{CD} $$
$$\boxed{C=b^TRb} \ \ \text{where} \ \ R:=Bb^T \ \ \text{and} \ \ C:=b^TB \tag{1}$$
Identity (1) is verified immediately due to the fact that $b^Tb=I$.
Explanations for the formulas of $R$ and $C$ in (1):
These three relationships can be gathered into the unique relationship $Rb=B$, itself equivalent (by using $bb^T=I$) to $R=Bb^T$.
where entry $C_{ij}$ of matrix $C$ is dot product
$$\vec{b_i} . \vec{B_j}=\|\vec{b_i}\|\|\vec{B_j}\|\cos(\theta_{ij})=\cos(\theta_{ij}),$$