In tetrahedron $SABC$, the circumcircles of faces $SAB$, $SBC$, and $SCA$ each have radius $108$. The inscribed sphere of $SABC$, centered at $I$, has radius $35.$ Additionally, $SI = 125$. Let $R$ be the largest possible value of the circumradius of face $ABC$. Given that $R$ can be expressed in the form $\sqrt{\frac{m}{n}}$, where $m$ and $n$ are relatively prime positive integers, find $m+n$.
(Alex Zhu)
Note: Hundreds of people/teams tried this problem and 0 people got this correct. One can find the solution to this problem online but the point of this thread is to discuss this amazing problem and see some nice StackExchange solutions!
Solution $-$ Victor Wang
Let $r=35$, $I$, $\omega$, $u$, $O$, and $\Omega$ be the inradius, incenter (given), insphere, circumradius, circumcenter, and circumsphere of $SABC$, respectively. Let $s=SI=125$; $Q$ be the reflection of $S$ over $O$; $F$ be the foot from $S$ to plane $ABC$; $h=SF$ be the length of the $S$-altitude; $I_A$, $I_B$, $I_C$ be the feet from $I$ to $SBC$, $SCA$, $SAB$, respectively; $O_A$, $O_B$, $O_C$, $O_S$ be the circumcenters of triangles $SAB$, $SBC$, $SCA$, $ABC$, respectively; and $a=108$ be the common circumradius of triangles $SAB$, $SBC$, $SCA$. For convenience, define $v = SI_A = SI_B = SI_C = \sqrt{s^2-r^2} = 120$.
First, we note that $OO_A^2 = OO_B^2 = OO_C^2 = u^2-a^2$ by the Pythagorean theorem, so $O$ is equidistant from the three planes $SAB$, $SBC$, $SCA$.
Taking the cross section formed by $\triangle{OO_BO_C}$, we see that $O$ lies on one of the two planes bisecting the angle formed by planes $SAB$ and $SAC$: either the interior one or the exterior one (these correspond naturally to the two-dimensional interior and exterior angle bisectors). Now assume for the sake of contradiction that $O$ lies on the exterior bisector (which is perpendicular to the interior one), and WLOG suppose plane $SAB$ separates $O$ and $C$. Since $O$ is the center of $\Omega=(SABC)$, we then have $a = R_{SAC} < R_{SAB} = a$, a contradiction. Thus $O$ must lie on the interior angle bisector $P_A$ of $\angle{(SAB,SAC)}$.
If we similarly define $P_B,P_C$, then $S,O,I\in P_A\cap P_B\cap P_C$ (as $I$ lies completely inside $PABC$). But the intersection of three distinct planes can only form a point, a line, or the empty set, so $S,O,I$ must be collinear. Let $\epsilon$ equal $+1$ if $O,S$ are on the same side of plane $ABC$ and $-1$ otherwise. Clearly $O$ cannot be higher than $S$ (relative to plane $ABC$), since the line $SI = SO$ intersects the interior of $\triangle{ABC}$. In particular, $S$ and $Q$ must lie on opposite sides of $ABC$. (*) Using similar triangles and the Pythagorean theorem, we get $$ \begin{align*} \frac{h-\epsilon OO_S}{h-r} = \frac{SO}{SI} = \frac{SO_A}{SI_A} \implies \frac{h-\epsilon\sqrt{u^2-R^2}}{h-r} = \frac{u}{s} = \frac{a}{v}.\tag{1} \end{align*}$$It immediately follows that $u = as/v = 225/2$.
We will now compute $h$ using inversion (this is natural because several circles and spheres in our diagram pass through $S$, and when we invert about $X$, the circumcenter of $\triangle{XYZ}$ is simply mapped to the reflection of $X$ over $Y'Z'$). Invert about $S$ with radius $\alpha$, and let $J_A,J_B,J_C$ be the midpoints of $I'_A,I'_B,I'_C$, respectively. Observe that $SQ$ is a diameter of $\Omega$, so $\angle{SQ'A'}=\angle{SQ'B'}=\angle{SQ'C'} = 90^\circ$, whence $Q'$ is the foot from $S$ to $A'B'C'$ (the spheres with diameters $SA'$, $SB'$, $SC'$ intersect at $S$ and its reflection over the plane formed by their centers). But $J_A$ is the foot from $S$ to $B'C'$, so $Q'P$ is minimized over $P\in B'C'$ at $P=J_A$ (by the Pythagorean theorem) and $J_A$ is the foot from $Q'$ to $B'C'$. On the other hand, $SJ_A=SJ_B=SJ_C = \frac{\alpha^2}{2SI_A}$ implies $Q'J_A = Q'J_B = Q'J_C$, whence $\triangle{A'B'C'}$ has incircle $(J_AJ_BJ_C)$ and inradius $$\begin{align*} t = Q'J_A = \sqrt{\left(\frac{\alpha^2}{2SI_A}\right)^2 - \left(\frac{\alpha^2}{2SO}\right)^2} = \frac{\alpha^2}{2}\sqrt{\frac{1}{a^2}-\frac{1}{u^2}} = \frac{\alpha^2}{2}\frac{r}{as}.\tag{2} \end{align*}$$(Indeed, according to (*), $Q'$ cannot be an excenter of $\triangle{A'B'C'}$, since $Q'$ lies on the same side of planes $SB'C'=SBC,SC'A'=SCA,SA'B'=SAB$ as $A',B',C'$, respectively.)
It remains to analyze $F'$. $\angle{SA'F'} = \angle{SB'F'} = \angle{SC'F'} = 90^\circ$, so $A',B',C'$ lie on the sphere $\Gamma$ with diameter $SF'$. Let $K$ be the midpoint of $SF'$ and $L$ be the foot from $K$ to $A'B'C'$; then $K$ is the center of $\Gamma$ and $L$ is the circumcenter of $\triangle{A'B'C'}$. If $x=SK$ denotes the radius of $\Gamma$, then $2x = SF' = \alpha^2/SF = \alpha^2/h$.
Fortunately, $Q'$, the foot of $S$, lies in the interior of $\triangle{A'B'C'}$, so $K$ cannot be higher than $S$ (relative to plane $A'B'C'$); hence $T$ lies on ray $\overrightarrow{SQ'}$. Since the (acute) angle $\theta$ between lines $SF'=SF$ and $SQ'=SQ$ is fixed under inversion, we get $$ \frac{Q'L}{x} = \frac{TK}{SK} = \sin\theta = \frac{\sqrt{SI^2 - (h-r)^2}}{SI} = \frac{\sqrt{s^2-(h-r)^2}}{s} $$and $$ KL = TQ' = \lvert SQ' - ST \rvert = \left\lvert \frac{\alpha^2}{2u} - SK\cos\theta \right\rvert = \left\lvert \frac{xh}{u} - x\frac{h-r}{SI} \right\rvert = x\left\lvert \frac{h}{u} - \frac{h-r}{s}\right\rvert. $$Plugging the previous two equations and (2) into Euler's formula (for $\triangle{A'B'C'}$) yields $$\begin{align*} x^2 - x^2\frac{(h-r)^2}{s^2} = x^2\frac{s^2-(h-r)^2}{s^2} = Q'L^2 &= R_{A'B'C'}^2 - 2tR_{A'B'C'} \\ &= (x^2 - KL^2) - \alpha^2\frac{r}{as}\sqrt{x^2 - KL^2} \\ &= x^2 - x^2\left(\frac{h}{u} - \frac{h-r}{s}\right)^2 - 2xh\frac{r}{as}x\sqrt{1 - \left(\frac{h}{u} - \frac{h-r}{s}\right)^2}. \end{align*}$$Adding $-x^2$ to both sides, substituting $as=uv$, and multiplying both sides by $-x^{-2}u^2s^2$, we get $$\begin{align*} u^2(h-r)^2 &= (sh - u(h-r))^2 + \frac{2hrs}{v}\sqrt{u^2s^2 - (sh - u(h-r))^2} \\ -s^2h^2 + 2shu(h-r) &= \frac{2hrs}{v}\sqrt{u^2s^2 - (sh - u(h-r))^2} \\ v(-sh+2u(h-r))& = 2r\sqrt{u^2s^2 - (sh - u(h-r))^2} \\ (s^2-r^2)[h^2(2u-s)^2 - 4h(2u-s)ur + 4u^2r^2] &= 4r^2 [-h^2(s-u)^2 - 2h(s-u)ur - u^2(r^2-s^2)] \\ v^2[h(2u-s)^2 - 4(2u-s)ur] &= 4r^2 [-h(s-u)^2 - 2(s-u)ur], \end{align*}$$whence $$\begin{align*} h = 4ur \frac{v^2(2u-s) - 2r^2(s-u)}{v^2(2u-s)^2 + 4r^2(s-u)^2} &= 4\frac{225}{2}35\frac{120^2(225-125) - (35)^2(2\cdot125-225)}{120^2(225-125)^2 + (35)^2(2\cdot125-225)^2} \\ &= 70\cdot15^2\frac{120^2\cdot10^2-35^2\cdot5^2}{120^2\cdot10^4+35^2\cdot5^4} \\ &= 70\cdot3^2\frac{24^2\cdot2^2-7^2}{24^2\cdot2^4+7^2} \\ &= \frac{284130}{1853}. \end{align*}$$We can find $\epsilon$ now (actually, $h>u$, so we trivially have $\epsilon=+1$), but this is unnecessary: plugging $h$ into (1) yields $$\begin{align*} R^2 &= u^2 - \left(h-\frac{a}{v}(h-r)\right)^2 \\ &= \frac{225^2}{2^2} - \left(\frac{284130}{1853}\left(1-\frac{108}{120}\right)+\frac{108\cdot35}{120}\right)^2 \\ &= \frac{9^2\cdot25^2}{2^2} - \left(\frac{28413}{1853}+\frac{9\cdot7}{2}\right)^2 \\ &= \frac{9^2\cdot25^2}{2^2} - \left(\frac{28413}{1853}+\frac{9\cdot7}{2}\right)^2 \\ &= \left(9^2 - \frac{28413}{1853}\right)\left(9\cdot16 + \frac{28413}{1853}\right) \\ &= \frac{121680\cdot295245}{1853^2} \\ &= \frac{35925411600}{3433609}. \end{align*}$$Thus the answer is $35925411600+3433609 = \boxed{35928845209}$.
I posted this as a: