How to prove that $X, Y,$ and $Z$ are collinear if and only if $\cos A \cos B \cos C =\frac{-3}{8}$?

Question

Let $X, Y, Z$ be the reflections of $A, B$, and $C$ across the lines $BC, CA,$ and $AB$, respectively. How to prove that $X, Y,$ and $Z$ are collinear if and only if $\cos A \cos B \cos C =\frac{-3}{8}$? for example α=120, β=γ=30.

Answer 1

In order to obtain manageable formulas we have to destroy the inherent symmetries of the problem.

Represent $A$, $B$, $C$ as complex numbers with $A$ at the origin and $B\vee C$ horizontal as in your figure. Then, up to similarity, we have $$A=0,\quad B=-\sin\gamma\> e^{-i\beta},\quad C=\sin\beta\> e^{i\gamma}$$ and $$X=2i\>\sin\beta\>\sin\gamma,\quad Y=-\sin\gamma\>e^{i(\beta+2\gamma)},\quad Z=\sin\beta\>e^{-i(2\beta+\gamma)}\ .$$ The three points $X$, $Y$, $Z$ are collinear iff $${X-Y\over Z-X}\in{\mathbb R},\qquad{\rm resp.},\qquad(X-Y)(\bar Z-\bar X)\in{\mathbb R}\ .$$ This amounts to $$\sin\gamma\bigl(2i\sin\beta+e^{i(\beta+2\gamma)}\bigr) \sin\beta \bigl(e^{i(2\beta+\gamma)}+2i\sin\gamma\bigr)\in{\mathbb R}\ .\tag{1}$$ Here the factor $\sin\beta\sin\gamma>0$ can be omitted, so that $(1)$ can be rewritten in real terms as $$\cos(\beta+2\gamma)\bigl(\sin(2\beta+\gamma)+2\sin\gamma\bigr)+\bigl(2\sin\beta+\sin(\beta+2\gamma)\bigr)\cos(2\beta+\gamma)=0\ .\tag{2}$$ Now Mathematica found out for me that the left hand side of $(2)$ can be factored into $$\sin(\beta+\gamma)\bigl(2\cos(2\beta+2\gamma)+2\cos(2\beta)+2\cos(2\gamma)-1\bigr)\ .$$ Since $\sin(\beta+\gamma)\ne0$ we may conclude that $$\cos(2\alpha)+\cos(2\beta)+\cos(2\gamma)={1\over2}\ ,$$ which according to @Aretino 's comment is equivalent to your claim.