An isosceles triangle $\triangle ABC$ is given with leg $AC=5$ and $R=\dfrac{25}{6}$ of the circumcircle. Find the base of the triangle.
This was my first sketch. The triangles $AHC$ and $OMC$ are similar, so $\dfrac{CH}{CM}=\dfrac{AC}{CO} \Leftrightarrow \dfrac{CO+OH}{CM}=\dfrac{AC}{CO}.$ When I plug with the values, I get a negative number for $OH$. I realised that this is possible if the triangle is obtuse. How should I notice this from the beginning? I have not studied trigonometry.
On
For an acute triangle where the two shortest sides are $a$ and $b$, the inradius is $r$, and the circumradius is $R$, the inequality $R+r < \frac{a+b}{2}$ is satisfied. So in this case, if the triangle were acute, you would need $\frac{25}{6} + r < \frac{5+5}{2}$, i.e. $r < \frac{5}{6}$, which is really small, and you can see from the diagram that it's quite unlikely.
Since in the standard notation $$\frac{abc}{4S}=R,$$ we obtain: $$\frac{5\cdot5\cdot c}{4\cdot\frac{c\sqrt{25-\frac{c^2}{4}}}{2}}=\frac{25}{6}$$ or $$\sqrt{25-\frac{c^2}{4}}=3,$$ which gives $$c=8.$$
We'll prove that $$\frac{abc}{4S}=R.$$
Indeed, let $\Phi$ be our circle and $CO\cap\Phi=\{C,D\}$.
Thus, $$\Delta ACH\sim\Delta DCB,$$ which gives $$\frac{CH}{BC}=\frac{AC}{CD}$$ or $$\frac{CH}{a}=\frac{b}{2R}$$ or $$CH=\frac{ab}{2R}.$$ Id est, $$\frac{abc}{4S}=\frac{abc}{4\cdot\frac{c\cdot\frac{ab}{2R}}{2}}=R.$$