I am unable to show - without expanding, by using determinant properties - that $$\det\begin{bmatrix} 1 &\cos a &\cos b\\ \cos a &1 &\cos(a+b) \\ \cos b &\cos(a+b) &1 \end{bmatrix}=0$$
I am using trigonometric identities to solve this but I don't understand what would be the next step.
This determinant is clearly zero.
The matrix is the Gram matrix of the three unit vectors $(1,0)$, $(\cos a,\sin a)$ and $(\cos b,-\sin b)$ in the plane. Your matrix equals $AA^T$ where $$A=\pmatrix{1&0\\\cos a&\sin a\\\cos b&-\sin b}$$ and so it is singular.