Testing whether the circumcenter of a cyclic quadrilateral lies inside it

Question

For a triangle with sides $a, b, c$ (where $c$ is the biggest side) there is a simple check to see whether it's circumcenter lies inside of it:

$$a^2 + b^2 < c^2$$

Is there such an inequality for a cyclic quadrilateral, given its side lengths $a, b, c, d$ (with longest side $d$)?

Can this be generalized to a cyclic convex $n$-gon?

Answer 1

The circumcenter of a triangle is inside the triangle if and only if $$ (a^2+b^2-c^2)(c^2+a^2-b^2)(b^2+c^2-a^2)\gt0\tag{1} $$

---

The diagonal with sides $a$ and $b$ on one side and $c$ and $d$ on the other is $$ e^2=\frac{\frac{a^2+b^2}{ab}+\frac{c^2+d^2}{cd}}{\frac1{ab}+\frac1{cd}}\tag{2} $$ Then the circumcenter is inside the quadrilateral if and only if $$ \hspace{-10pt}\small(a^2+b^2-e^2)(e^2+a^2-b^2)(b^2+e^2-a^2)(c^2+d^2-e^2)(e^2+c^2-d^2)(d^2+e^2-c^2)\lt0\tag{3} $$ or $$ a^2+b^2=c^2+d^2\tag{4} $$ Note that $(4)$ implies equality in $(3)$.

Answer 2

Let $a, b, c, d$ be the lengths of the sides with $a$ being longest or tied for longest (we will eventually see that if there is a tie then the circumcenter is forced inside the polygon). Interchanging the order of the sides has no effect on the circumcircle and so WLOG the $a$ and $b$ sides are taken to be adjacent to each other.

Draw the diagonal that separates the $a, b$ sides from the $c, d$ sides. Let $\theta$ be the angle between the $a$ and $b$ sides, $\theta'$ be the angle between the $c$ and $d$ sides. Apply the Law of Cosines to the two triangles formed by the diagonal:

$\cos(\theta)=(a^2+b^2-x^2)/(2ab)$

$\cos(\theta')=(c^2+d^2-x^2)/(2cd)$

The two angles are supplementary, therefore their cosines sum to zero and the right sides of the above equations must do the same. Form that Sun and solve for $x^2$:

$x^2=(ab(c^2+d^2)+cd(a^2+b^2))/(ab+cd)$

The circumcenter is outside the quadrilateral when it is on tge opposite side of the $a$ side from the diagonal we drew above. That condition is:

$a^2>x^2+b^2$

Substitute for x, work through the algebra and obtain this result for the circumcenter lying outside the quadtilateral:

$a^3-a(b^2+c^2+d^2)-2bcd>0$

Note that if $b$,$c$, or $d$ matches $a$, the left side is forced to be negative, pushing the circumenter inside the polygon.