Elmo is now learning olympiad geometry. In triangle $ABC$ with $AB\neq AC$, let its incircle be tangent to sides $BC$, $CA$, and $AB$ at $D$, $E$, and $F$, respectively. The internal angle bisector of $\angle BAC$ intersects lines $DE$ and $DF$ at $X$ and $Y$, respectively. Let $S$ and $T$ be distinct points on side $BC$ such that $\angle XSY=\angle XTY=90^\circ$. Finally, let $\gamma$ be the circumcircle of $\triangle AST$.
(a) Help Elmo show that $\gamma$ is tangent to the circumcircle of $\triangle ABC$.
(b) Help Elmo show that $\gamma$ is tangent to the incircle of $\triangle ABC$.
Very Hard problem ..
My Progress: WLOG $AB<AC$ . Define $N=BC\cap AY $.
Claim: $\angle ASX=\angle XST$
Proof: Since $\angle XSY=90$ , by lemma 9.18 from EGMO it's enough to show that $(A,N;X,Y)=-1$ .
Now, note that $AEFD$ is a harmonic quad . So $(A,D;E,F)=-1$ . So projecting through $D$ on $AI$, we get $(A,DD\cap AI=N;DE\cap AI=X,DF\cap AI=Y )=-1$ . So $\angle ASX=\angle XST$.
Similarly , we can get $\angle ATX=\angle XTN$
So we get $X$ and $Y$ as the incentre and $A$-excentre of $\Delta AST$ respectively .
Also we get $AS$ and $AT$ isogonals [ since $X$ is the incentre and $X\in AI$ ]
Now, let $S'=AS\cap (ABC)$ , $T'=AT\cap (ABC)$ . By angle chase , we get $ST||S'T'$ and hence $AST~AS'T'$ and by homothety , we get $(AST)$ and $(ABC)$ tangent.
This proves part $A$.
For Part $B$: I applied inversion $\psi$ centered at $A$ with radius $\sqrt{AX\cdot AY}$ radius followed by reflection about the angle bisector of $\angle AST$.
Note that $\psi:X\leftrightarrow Y$, $\psi:S\leftrightarrow T$ and $\psi:BC\leftrightarrow (AST)$ .
Let us assume that $\angle AXB=\angle AYC=90$ [ I haven't got the proof of this , but it looks very true]
Now, let $I-A$ be the $A$-excentre of $\Delta ABC$ , $F^*$ be touch point of $A$-excircle to line $AC$, $E^*$ be touch point of $A$ -excircle to line $AB$, And $K^*$ be touch point of $A$-excircle to $BC$.
Note that $K^*,C,F^*,I-A$ is cyclic .
But it's known that $\angle K^*CF^*=180-C\implies \angle CK^*F^*=\angle CF^*K^*=C/2$ . So $\angle XYF*=90+C/2$ .
Also note that $\Delta DXC$ is isosceles , hence $\angle XEC=90-C/2$ .
Hence $XEF^*Y$ is cyclic. Similarly we can get $XFE^*Y$ is cyclic [$BK*I-AE*$ is cyclic]
Hence $AX\cdot AY=AE\cdot AF^*=AF\cdot AE^*$ .
So $\psi:E\leftrightarrow E^*$ and $\psi:F\leftrightarrow F^*$ .
Define $D^*$ as the inverse image of $D$ by $\psi$.
Note that $(D^*E^*F^*)$ will be a circle tangent to $AB$ at $E^*$, tangent to $AC$ at $F^*$ and $(AST)$ at $D^*$ .
If I am able to show that $(D^*E^*F^*)$ is the $A$-excircle , then I am done . So basically, I need to prove that
Show that $A-$excircle is tangent to $(AST)$
After Proofing the above statement , we get that $\psi:$ incircle of $ABC$ $\leftrightarrow$ $ A$-excircle of $ABC$. Now since $A$-excircle touches $BC$ at $K^*$ , and inversion preserves tangency , the inverted images will also be tangent to each other i.e incircle of $ABC$ is tangent to $(AST)$ and we will be done ..
If possible can someone post solution using the way I have proceeded ( using $\sqrt{AX\cdot AY}$) ? Thanks in advance!
Here's the ggb link for the diagram: https://www.geogebra.org/geometry/xzzqzmuh
Wonderful approach!
Filling the blanks you left...
Proof. Note that $\angle IED +\angle IXE = \angle AIE=\dfrac{\angle B+\angle C}{2}$ and as $$\angle IED = \dfrac{\angle C}{2}\implies \angle IXE =\dfrac{\angle B}{2}=\angle DXY=\angle IBD\implies X\in\odot(BID)\implies \angle IXB=90^\circ$$ and similarly, we get $\angle AYC=90$ completing the proof.
So basically, all you wanna show is that the incircle maps to $A$-excircle under the $\sqrt{AX\cdot AY}$ inversion along with reflection about angle bisector $\angle SAT$. As you have already shown that $\Psi: \{F, E\}\leftrightarrow \{F',E'\} $ so its enough to show that there is at least one point $P\in \odot(I), P\not\in\{E,F\}$ such that $\Psi(P)\in\odot(I_A)$. As you have already shown that $\{AS,AT\}$ are isogonal, we know that angle bisector of $\angle SAT$ is nothing but $AI$. As reflection of $A$-excircle about $AI$ is $A$-excircle, its enough to show that there is a point $P\in\odot(I)$ (other than $E,F$) such that it maps to some point on $A$-excircle after inverting about the circle centered at $A$ with radius $\sqrt{AX\cdot AY}$. As you found that $XYFE'$ is cyclic, by power of point, we have $AX\cdot AY = AF\cdot AE'$. Let $D$-antipode in $\odot(I)$ be $M$. It is well known that $A-M-K'$ are collinear. Let $W= AK'\cap \odot(I_A)$, $W\neq K'$. Note that the homothety centered at $A$ that maps incircle to $A$-excircle tells us that $MF\|K'E'$ and thus, $$\angle AFM=\angle AE'K'=\angle E'WK'\implies MWE'F$$ is cyclic. Thus, $$AX\cdot AY = AE'\cdot AF= AM\cdot AW$$ and thus, $M$ maps to $W$ under the $\sqrt{AX\cdot AY}$ inversion. Thus, such a $P$ exsist and hence, under the $\sqrt{AX\cdot AY}$ inversion, the incircle gets maped to $A$-excircle. Thus, $(D^∗E^∗F^∗)$ is the $A$-excircle and we are done!$\tag*{$\blacksquare$}$