Given the symmetrical matrix A and the diagonal matrix D:
$$A = \begin{pmatrix} 2 & -\frac{1}{2}\sqrt{3} \\ -\frac{1}{2}\sqrt{3} & 1 \\ \end{pmatrix},$$
$$D = \begin{pmatrix} \frac{1}{2} & 0 \\ 0 & \frac{5}{2} \\ \end{pmatrix}$$
For the matrix to be orthogonal, its determinant has to be either $-1$ or $1$, but I don't know how to proceed.
You want an orthogonal $Q$ such that $A=QDQ^\top$.
As others have hinted, note that $D$ contains the eigenvalues of $A$. The corresponding eigenvectors are the columns of $Q$. To see this, suppose $q_1$ is the first column of $Q$. Then \begin{align} Aq_1 &= QDQ^\top q_1\\ &= QD\begin{bmatrix}1\\0\end{bmatrix} & \text{orthogonormal columns}\\ &= Q \frac{1}{2} \begin{bmatrix}1\\0\end{bmatrix}\\ &= \frac{1}{2} q_1. \end{align} Similarly, $Aq_2=\frac{5}{2} q_2$.
So, to finish the problem, we just need to find two eigenvectors of $A$, scale them so that they have unit norm, and then $Q$ will have these two vectors as its columns.
The eigenvectors of $A$ can be found by finding a vector in the nullspace of $A-\lambda I$ for $\lambda=1/2$ and $\lambda=5/2$.