$\mathrm{OABC}$ is a tetrahedron with $\overline{\mathrm{OA}}=1$.
There is a point $P$ on $\triangle \mathrm{ABC}$ such that $\cos^2 \alpha+\cos^2 \beta + \cos^2 \gamma = \frac{11}{6}$ where $\alpha=\angle\mathrm{AOP}$, $\beta=\angle\mathrm{BOP}$ and $\gamma=\angle\mathrm{COP}$.
Describe the trace of such $P$.
We can solve this problem by using the following lemma:
Let's imagine the tetrahedron with four vertices $O(0,0,0)$, $A(a,b,c)$, $B(b,c,a)$ and $C(c,a,b)$.
Since $\overline{\mathrm{OA}}=\overline{\mathrm{OB}}=\overline{\mathrm{OC}}=\overline{\mathrm{AB}}=\overline{\mathrm{BC}}=\overline{\mathrm{CA}}$, we have
$a^2+b^2+c^2=(a-b)^2+(b-c)^2+(c-a)^2\Rightarrow a^2+b^2+c^2=2(ab+bc+ca)$.
And since $\vec{\mathrm{OA}}=(a,b,c)$, $\vec{\mathrm{OP}}=(x,y,z)$, we have $\cos^2 \alpha=\left(\frac{\vec{\mathrm{OA}} \cdot \vec{\mathrm{OP}}}{\vert \vec{\mathrm{OA}} \vert \vert\vec{\mathrm{OP}}\vert}\right)^2=\frac{(ax+by+cz)^2}{(a^2+b^2+c^2)(x^2+y^2+z^2)}$.
WLOG $\cos^2 \beta=\frac{(bx+cy+az)^2}{(a^2+b^2+c^2)(x^2+y^2+z^2)}$, $\cos^2 \gamma=\frac{(cx+ay+bz)^2}{(a^2+b^2+c^2)(x^2+y^2+z^2)}$.
Then by the given equation, $\frac{(ax+by+cz)^2+(bx+cy+az)^2+(cx+ay+bz)^2}{(a^2+b^2+c^2)(x^2+y^2+z^2)}=1+\frac{2(ab+bc+ca)(xy+yz+zx)}{(a^2+b^2+c^2)(x^2+y^2+z^2)}=\frac{11}{6}$.
By substituting $a^2+b^2+c^2=2(ab+bc+ca)$, we obtain $5(x^2+y^2+z^2)=6(xy+yz+zx)$.
Now, observe that $P$ is on the plane $x+y+z=a+b+c$.
So $2(xy+yz+zx)=(x+y+z)^2-(x^2+y^2+z^2)=(a+b+c)^2-(x^2+y^2+z^2)$ and $8(x^2+y^2+z^2)=3(a+b+c)^2$.
As a conclusion, the trace of $P$ is a circle which is an intersection between the sphere $x^2+y^2+z^2=\frac{3}{8}(a+b+c)^2$ and the plane $x+y+z=a+b+c$.
I think it is a quite simple and nice solution, but I do not want to use such analytic methods.
I tried many times to solve this only with euclidean geometry, but I could not find a better strategy.
Would you help me?
Assuming $OABC$ is a regular tetrahedron of edge length 1.
Place point O at the origin $(0,0,0)$, and let face $ABC$ be parallel to the $xy$ plane, then points $A, B, C$ can be taken as
$A = (\sin \theta , 0, \cos \theta )$
$B = (\sin \theta \cos \phi, \sin \theta \sin \phi, \cos \theta) $
$C = (\sin \theta \cos(- \phi) , \sin \theta \sin(-\phi) , \cos \theta ) $
where $\cos \theta = \sqrt{\dfrac{2}{3}} $ and $\sin \theta = \dfrac{1}{\sqrt{3}} $
and $\phi = \dfrac{2 \pi}{3} $
Hence, $A = (\dfrac{1}{\sqrt{3}} , 0, \sqrt{\dfrac{2}{3}} )$
$B = (-\dfrac{1}{2 \sqrt{3}}, \dfrac{1}{2}, \sqrt{\dfrac{2}{3}} ) $
$C = (-\dfrac{1}{2 \sqrt{3}} , -\dfrac{1}{2} , \sqrt{\dfrac{2}{3}}) $
A point $P$ on $\triangle ABC$ can be parameterised as follows
$P = A + t (B - A) + s (C - A) $ where $ 0 \le t \le 1 $ and $ 0 \le s \le 1 - t $
therefore,
$ P = (1 - t - s) A + t B + s C $
Hence,
$\cos AOP = \dfrac{ A \cdot P }{ (1) |P| } $
From the expresssion of $P$ and the fact that all faces of the tetrahedron are equilateral triangles of side length (1), then
$A \cdot P = (1 - t - s) + 0.5 t + 0.5 s = 1 - 0.5 t - 0.5 s $
$B \cdot P = 0.5 (1 - t - s) + t + 0.5 s = 0.5 + 0.5 t $
$C \cdot P = 0.5 (1 - t - s) + 0.5 t + s = 0.5 + 0.5 s$
Further, we know that
$\begin{equation} \begin{split} P \cdot P &= ((1 - t - s) A + t B + s C)\cdot ((1 - t- s)A + t B + sC) \\ &= (1 - t - s)^2 + t^2 + s^2 + ( t(1 - t - s) + s (1 - t -s) + t s)\\ &= 1 + t^2 + s^2 - 2 t - 2 s + 2 t s + t + s - 2 t s + t s \\ &= 1 + t^2 + s^2 - t - s + t s\\ \end{split} \end{equation}$
Hence the given condition can be expressed in terms of $t $ and $ s$ as follows:
$6[(1 - 0.5 t - 0.5 s)^2 + (0.5 + 0.5 t)^2 + (0.5 + 0.5 s)^2 ] = 11 (1 + t^2 + s^2 - t - s + t s)$
Expanding,
$ 6 ( 1.5 + 0.5 t^2 + 0.5 s^2 - 0.5 t - 0.5 s + 0.5 t s ) = 11 (1 + t^2 + s^2 - t - s + t s) $
Hence,
$ 9 + 3 t^2 + 3 s^2 - 3 t - 3 s + 3 t s = 11 + 11 t^2 + 11 s^2 - 11 t - 11 s + 11 t s $
And finally,
$ 8 t^2 + 8 s^2 + 8 ts - 8 t - 8 s + 2 = 0 $
Dividing through by $2$,
$ 4 t^2 + 4 s^2 + 4 t s - 4 t - 4 s + 1 = 0$
Which is the equation of a conic, define the vector $r = [t, s]^T $ then in matrix form, the equation of the conic becomes,
$ r^T Q r + b^T r + c = 0$
with
$Q = \begin{bmatrix} 4 && 2 \\ 2 && 4 \end{bmatrix}$
$b = \begin{bmatrix} -4 \\ -4 \end{bmatrix}$
$ c = 2$
To diagonalize $Q$, take $\theta = \dfrac{\pi}{4} $, then
$Q = R D R^T $, with
$D_11 = 4 (1/2) + 4 (1/2) + 2 (1) = 6 $
$D_22 = 4 (1/2) + 4 (1/2) - 2 (1) = 2$
The center of the conic (in terms of $(t, s)$ ) is at
$r_0 = - \frac{1}{2} Q^{-1} b = - \frac{1}{2} \frac{1}{12} \begin{bmatrix} 4 && -2 \\ -2 && 4 \end{bmatrix} \begin{bmatrix} -4\\-4 \end{bmatrix} = \dfrac{1}{3} \begin{bmatrix} 1 \\ 1 \end{bmatrix} $
Now, the equation is,
$ (r - r_0)^T R D R^T (r - r_0) - r_0^T R D R^T r_0 + c = 0 $
And we have
$ R^T r_0 = \frac{\sqrt{2}}{3} \begin{bmatrix} 1 \\ 0 \end{bmatrix} $
Hence, $r_0^ R D R^T r_0 = 6 (2 / 9) = 4/3 $
So that, now the equation is finally,
$ (r - r_0)^T R D R^T (r - r_0) -4/3 +1 = 0 $
which simplifies to,
$ (r - r_0)^T R D R^T (r - r_0) = 1/3 $
Multiplying through by 3, gives
$ (r - r_0)^T R (3 D) R^T (r - r_0) = 1 $
And the equation of an ellipse, we semi-major axis of $\dfrac{1}{\sqrt{6}}$ and semi-minor axis of $\frac{1}{3 \sqrt{2}}$.
Hence,
$r = r_0 + R D_1 u$
where $D_1 = \text{diag} \{ \dfrac{1}{3 \sqrt{2}}, \dfrac{1}{\sqrt{6}} \}$
and $u = [ \cos \psi, \sin \psi ] $. It follows that trace of $P$ can be written as
$P = A + t (B - A) + s (C - A) = A + G r = A + G (r_0 + R D_1 u) = A + G r_0 + G R D_1 u $
where $A = (\dfrac{1}{\sqrt{3}} , 0, \sqrt{\dfrac{2}{3}} )$
$B = (-\dfrac{1}{2 \sqrt{3}}, \dfrac{1}{2}, \sqrt{\dfrac{2}{3}} ) $
$C = (-\dfrac{1}{2 \sqrt{3}} , -\dfrac{1}{2} , \sqrt{\dfrac{2}{3}}) $
$G = \begin{bmatrix} B-A && C-A \end{bmatrix} = \begin{bmatrix} -\dfrac{\sqrt{3}}{2} && -\dfrac{\sqrt{3}}{2} \\ \dfrac{1}{2} && -\dfrac{1}{2} \\ 0 && 0 \end{bmatrix} $
$P = A + G r_0 + G R D_1 u $
With a little bit of number crunching, one finds that
$P = P_0 + P_1 u $
with
$P_0 = A + G r_0 = \begin{bmatrix} 0 \\ 0 \\ \sqrt{\dfrac{2}{3}} \end{bmatrix} $
$P_1 = G R D_1 = \begin{bmatrix} -\dfrac{1}{\sqrt{12}} && 0 \\ 0 && -\dfrac{1}{\sqrt{12}} \\ 0 && 0 \end{bmatrix} $
By inspection this is an equation of a circle of radius $\dfrac{1}{\sqrt{12}}$ lying in a plane parallel to the $xy$ plane with its center at $(0, 0, \sqrt{\dfrac{2}{3}})$