I have matrix $A=\begin{pmatrix} 0&-1&0\\1&0&0\\0&0&1\\\end{pmatrix}$ and matrix $B=\begin{pmatrix} 1&0&0\\0&\frac{1}{2}&-\frac{\sqrt{3}}{2}\\0&\frac{\sqrt{3}}{2}&\frac{1}{2}\\\end{pmatrix}$ and I want to find a continuous path between them. How do I do this?
I know that matrices in $SO(n)$ should be path connected but how do you actually construct such a path?
I'm not sure if just taking $\gamma(t)=A(t)+B(1-t)$ is going to work.
You can check that any two dimensional rotation matrix can be written as $$R_\theta = \begin{bmatrix}\cos\theta & -\sin\theta \\ \sin\theta & \cos\theta\end{bmatrix} = e^{\theta J} \quad \text{where} \quad J = \begin{bmatrix}0 & -1 \\ 1 & 0\end{bmatrix}.$$
Using this fact, we have $$R_{\pi/2} = \begin{bmatrix}0 & -1 \\ 1 & 0\end{bmatrix} = e^{(\pi/2)J} \quad \text{and} \quad R_{\pi/3} = \begin{bmatrix}0 & -1 \\ 1 & 0\end{bmatrix} = e^{(\pi/3)J}.$$
Block-diagonal matrices $M = \begin{bmatrix}M_1 & 0 \\ 0 & M_2\end{bmatrix}$ satisfy $e^M = \begin{bmatrix}e^{M_1} & \\ & e^{M_2}\end{bmatrix}$.
Therefore, $$A = \begin{bmatrix}0 & -1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1\end{bmatrix} = e^{C} \quad \text{where} \quad C = \begin{bmatrix} \tfrac{\pi}{2}J & \\ & 0 \end{bmatrix} = \dfrac{\pi}{2}\begin{bmatrix}0 & -1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 0\end{bmatrix},$$ and $$B = \begin{bmatrix}1 & 0 & 0 \\ 0 & \tfrac{1}{2} & -\tfrac{\sqrt{3}}{2} \\ 0 & \tfrac{\sqrt{3}}{2} & \tfrac{1}{2}\end{bmatrix} = e^{D} \quad \text{where} \quad D = \begin{bmatrix} 0 & \\ & \tfrac{\pi}{3}J \end{bmatrix} = \dfrac{\pi}{3}\begin{bmatrix}0 & 0 & 0 \\ 0 & 0 & -1 \\ 0 & 1 & 0\end{bmatrix}.$$
Finally, you can check that for any anti-symmetric real matrix $H$, $e^H$ is in $SO(n)$. Can you use these facts to connect $A = e^C$ to $B = e^D$ by a continuous path in $SO(3)$? You have the right idea to use something like $t \cdot (\text{start point}) + (1-t) \cdot (\text{end point})$, but you need to tie in the matrix exponential somehow to guarantee the path stays in $SO(3)$.