Let $O,A,B,C$ be points in space such that $\angle AOB=60^{\circ},\angle BOC=90^{\circ},\angle COA=120^{\circ}$ Let $\theta$ be the acute angle between the planes $AOB$ and $AOC$. Find $\cos\theta.$
Here is what I did:
I let $AO=OB=BA=1, OC=1$. Then we can compute $BC=\sqrt{2},AC=\sqrt{3}$. From here, I let $A=(0,0,0),B=(1,0,0),C=(\frac{1}{2},\frac{\sqrt{3}}{2},0),C=(1,0,\sqrt{2})$ (I found the coordinates of $C$ by noticing $\angle ABC=90^{\circ}$ since $AB^2+BC^2=AC^2$). Then I found the equation of $AOC$ to be $-\sqrt{6}x+\sqrt{2}y+\sqrt{3}z=0$ and the equation of $AOB$ to be $z=0$, so using the formula here, I got $\cos\theta=\frac{\sqrt{3}}{11}$, but the answer is $\frac{1}{3}$. Where did I mess up?
Also a solution without using coordinates would be very nice to see!
Your starting was right, so we have $$ O = (0,0,0)\quad A = (1,0,0)\quad B = {1 \over 2}\left( {1,\sqrt 3 ,0} \right) $$ for point C instead, we shall have $$ \left\{ \matrix{ \mathop {OB}\limits^ \to \, \cdot \;\mathop {OC}\limits^ \to = \cos \left( {90^\circ } \right) = 0 \hfill \cr \mathop {OA}\limits^ \to \, \cdot \;\mathop {OC}\limits^ \to = \cos \left( {120^\circ } \right) = - 1/2 \hfill \cr} \right. $$ and putting $\mathop {OC}\limits^ \to = \left( {x,y,z} \right)$ $$ \left\{ \matrix{ x + \sqrt 3 y = 0 \hfill \cr x = - 1/2 \hfill \cr} \right.\quad \to \quad \left\{ \matrix{ x = - 1/2 \hfill \cr y = {1 \over {2\sqrt 3 }} \hfill \cr} \right. $$ Now $z$ comes to be "free", but if you want $\left| {\mathop {OC}\limits^ \to } \right| = 1$ then $z = {{\sqrt 2 } \over {\sqrt 3 }}$ , so $$ \mathop {OC}\limits^ \to = \left( { - {1 \over 2},{1 \over {2\sqrt 3 }},{{\sqrt 2 } \over {\sqrt 3 }}} \right) $$ Once you get three points, i.e. two vectors, lying on a plane, the cross-product of the two vectors will give a vector normal to the plane. So $$ {\bf n} = \mathop {OA}\limits^ \to \, \times \,\mathop {OB}\limits^ \to = \left( {0,0,{{\sqrt 3 } \over 2}} \right)\quad {\bf m} = \mathop {OA}\limits^ \to \; \times \;\mathop {OC}\limits^ \to = \left( {0, - {{\sqrt 2 } \over {\sqrt 3 }}, - {1 \over {2\sqrt 3 }}} \right) $$ and the dot-product of the normal unit vectors to two planes will give you the cosine of their dihedral angle (apart the sign), thus $$ {{\bf n} \over {\left| {\bf n} \right|}} \cdot {{\bf m} \over {\left| {\bf m} \right|}} = \pm \cos \theta \quad \to \quad \left| {\left( { - {1 \over 4}} \right)\left( {{{\sqrt 3 } \over 2}\sqrt {\left( {{2 \over 3} + {1 \over {12}}} \right)} } \right)^{\, - 1} } \right| = {1 \over 4}{4 \over {\sqrt 3 \sqrt 3 }} = {1 \over 3} = \cos \theta $$