Suppose you're given the four vertices of a tetrahedron $ABCD$. You want to find the equation of the ellipsoid that is inscribed in it and tangent to three of its faces at $r_1, r_2, r_3$. How would you compute the equation of this inscribed ellipsoid?
My attempt: Is detailed in my answer below.
Hints, comments, and answers are highly appreciated.
First determine the vertex $V$ that is the intersection of the three faces that contain $r_1, r_2, r_3$. Using a affine transformation, transform the given tetrahedron, into a tetrahedron with vertices $(0,0,0), (1, 0, 0), (0, 1, 0), (0, 0, 1)$.
This affine transformation is
$ f(X) = T (X - V) \tag{1}$
To determine $T$, plug in the other vertices, other than $V$ in this equation. For example, if $V = A$ , then
$ e_1 = T (B - A) \tag{2}$
$ e_2 = T (C - A) \tag{3}$
$ e_3 = T (D - A) \tag{4}$
Thus,
$ T = \begin{bmatrix} B- A && C-A && D-A \end{bmatrix}^{-1} $
Then, compute the transformed tangency points:
$ r'_1 = T (r_1 - A) \tag{5}$
$ r'_2 = T (r_2 - A ) \tag{6}$
$ r'_3 = T (r_3 - A ) \tag{7}$
The equation of the ellipsoid in the transformed tetrahedron is
$ F(r) = (r - r_0)^T Q (r - r_0) = 1 \tag{8} $
The gradient vector is
$ \nabla_{r} F = 2 Q (r - r_0) \tag{9} $
Assuming the tangency points $r'_1 , r'_2, r'_3$ have been ordered such that $r'_1$ lies in the $YZ$ plane, $r'_2$ lies in the $XZ$ plane, and $r'_3$ lies in the $XY$ plane, then
$ Q (r'_1 - r_0 ) = - k_1 e_1 \tag{10}$
where $k_1 \gt 0$, and $e_k$ is the $k$-th standard basis vector of $\mathbb{R}^3$.
From $(10)$, we get
$ r'_1 - r_0 = - k_1 Q^{-1} e_1 \tag{11} $
Substituting this into the equation of the ellipsoid $(8)$, gives us
$ k_1 = \sqrt{ e_1^T Q^{-1} e_1 } \tag{12}$
Similarly, we will have
$ r'_2 - r_0 = - k_2 Q^{-1} e_2 \tag{13}$
with
$k_2 = \sqrt{ e_2^T Q^{-1} e_2 } \tag{14}$
and
$ r'_3 - r_0 = - k_3 Q^{-1} e_3 \tag{15} $
with
$ k_3 = \sqrt{ e_3^T Q^{-1} e_3 } \tag{16}$
Note that
$ r'_1 = (0, r_{1y}, r_{1z} ) , r'_2 = (r_{2x}, 0, r_{2z} ) , r'_3 = (r_{3x}, r_{3y}, 0) $
Using $i = e_1, j = e_2, k = e_3$, then from $(11) - (15)$, we get
$ r_{0x} = \sqrt{i^T Q^{-1} i} = r_{2x} + \dfrac{i^T Q^{-1} j}{ \sqrt{j^T Q^{-1} j}} = r_{3x} + \dfrac{i^T {Q^-1} k} { \sqrt{k^T Q^{-1} k}} \tag{17}$
$r_{0y} = r_{1y} + \dfrac{j^T Q^{-1} i}{ \sqrt{i^T Q^{-1} i}} = \sqrt{j^T Q^{-1} j} = r_{3y} + \dfrac{j^T Q^{-1} k}{\sqrt{k^T Q^{-1} k}} \tag{18}$
$r_{0z} = r_{1z} + \dfrac{k^T Q^{-1} i}{ \sqrt{i^T Q^{-1} i}} = r_{2z} + \dfrac{k^T Q^{-1} j}{ \sqrt{j^T Q^{-1} j} } = \sqrt{ k^T Q^{-1} k }\tag{19}$
Hence,
$r_{0x} = r_{2x} + \dfrac{i^T Q^{-1} j}{r_{0y}} = r_{3x} + \dfrac{i^T Q^{-1} k}{r_{0z}}\tag{20}$
$r_{0y} = r_{1y} + \dfrac{j^T Q^{-1} i}{ r_{0x}} = r_{3y} + \dfrac{j^T Q^{-1} k }{r_{0z}}\tag{21}$
$r_{0z} = r_{1z} + \dfrac{k^T Q^{-1} i}{ r_{0x}} = r_{2z} + \dfrac{k^T Q^{-1} j}{r_{0y}} \tag{22}$
Solving for $i^T Q^{-1} j$, $i^T Q^{-1} k$ , $j^T Q^{-1} k$, gives us
$i^T Q^{-1} j = (r_{0x} - r_{2x}) r_{0y} = (r_{0y} - r_{1y}) r_{0x} \tag{23} $
$i^T Q^{-1} k = (r_{0x} - r_{3x}) r_{0z} = (r_{0z} - r_{1z}) r_{0x} \tag{24}$
$j^T Q^{-1} k = (r_{0y} - r_{3y}) r_{0z} = (r_{0z} - r_{2z}) r_{0y} \tag{25}$
hence
$r_{2x} r_{0y} = r_{1y} r_{0x}\tag{26}$
$r_{3x} r_{0z} = r_{1z} r_{0x}\tag{27}$
$r_{3y} r_{0z} = r_{2z} r_{0y} \tag{28}$
It follows that the coordinates of $r'_1$, $r'_2$ and $r'_3$ must satisfy,
$-r_{1y} ( r_{3x} r_{2z} ) + r_{1z} (r_{2x} r_{3y}) = 0 \tag{29}$
i.e.
$- r_{3x} ( r_{1y} r_{2z} ) + r_{3y} (r_{1z} r_{2x}) = 0\tag{30}$
What this means is that the coordinates of $r_1, r_2, r_3$ are not totally independent. So one can specify $r_1$ and $r_3$ and the $x$ coordinate of $r_3$, and then compute its $y$ coordinate from $(30)$ as follows
$ r_{3y} = r_{3x} \left( \dfrac{ r_{1y} r_{2z} }{ r_{1z} r_{2x} } \right) \tag{31}$
It is important to note that we must have $r_1, r_2, r_3$ lying inside the respective triangular faces of the tetrahedron. This is guaranteed if
$ r_{1y} , r_{1z} , r_{2x}, r_{2z}, r_{3x}, r_{3y} \gt 0 \tag{32a} $
And
$ r_{1y} + r_{1z} \lt 1 \tag{32b}$
$ r_{2x} + r_{2z} \lt 1 \tag{32c}$
$ r_{3x} + r_{3y} \lt 1 \tag{32d}$
and
$(r_{0x}, r_{0y}, r_{0z}) = t d \tag{33}$
where
$d =\left( 1 , \dfrac{r_{1y}}{r_{2x}} , \dfrac{r_{1z}}{ r_{3x}} \right) \tag{34} $
Now to find t ( to determine $r_0$ and $Q^{-1}$ ) , note the tangency point $r'_4$ (yet unknown) satisfies
$n^T (r'_4 - (1, 0, 0) ) = 0\tag{35}$
and
$r'_4 - t d = \dfrac{Q^{-1} n}{\sqrt{ n^T Q^{-1} n }} \tag{36} $
where $n = (1,1,1)$
hence,
$n^T (r'_4 - t d) = 1 - t (d_x + d_y + d_z) = \sqrt{n^T Q^{-1} n} \tag{37}$
from which, by squaring both sides,
$1 - 2 t (d_x + d_y + d_z) + t^2 (d_x + d_y + d_z)^2 = n^T Q^{-1} n \tag{38}$
Note that $n = (1,1,1) = i + j + k $, hence,
$n^T Q^{-1} n = i^T Q^{-1} i + j^T Q^{-1} j + k^T Q^{-1} k + 2 i^T Q^{-1} j + 2 i^T Q^{-1} k + 2 j^T Q^{-1} k \tag{39}$
but
$ i^T Q^{-1} j = (r0y - r1y) r0x = t d_x ( t d_y - r_{1y} ) \tag{40}$
$i^T Q^{-1} k = (r_{0z} - r_{1z}) r_{0x} = t d_x ( t d_z - r_{1z} ) \tag{41}$
$j^T Q^{-1} k = (r_{0z} - r_{2z}) r_{0y} = t d_y ( t d_z - r_{2z} ) \tag{42}$
Substituting these, and cancelling equal terms, results in,
$1 - 2 t (d_x + d_y + d_z) + 2 t d_x r_{1y} + 2 t d_x r_{1z} + 2 t d_y r_{2z} = 0 \tag{43}$
So that
$ t = \dfrac{1 }{ 2 (d_x + d_y + dz) - 2 d_x ( r_{1y} + r_{1z} ) - 2 d_y r_{2z} } \tag{44}$
Using this unique value of $t$ , we can determines the center $r_0$ of the ellipsoid using $(33)$.
And in addition, having obained the center $r_0$, we can use equations $(17)-(19)$ and $(23)-(25)$ to determine all the entries of $Q^{-1}$.
Finally, invert $Q^{-1}$ to obtain $Q$.
This fully specifies the unique inscribed ellipsoid in the transformed tetrahedron.
The equation we've obtained is $ (r' - r_0)^T Q (r' - r_0) = 1 $, but from $(1)$, we have $ r' = T (r - V) $, therefore,
$ (T (r - V) - r_0 )^T Q (T (r - V) - r_0) = 1 $
$r$ is the coordinate of a point in the original space. This equation becomes
$ (r - (V + T^{-1} r_0) )^T T^T Q T ( r - (V + T^{-1} r_0) ) = 1 \tag{45}$
Here's an example for a tetrahedron with vertices $(0,0,0), (5, 0, 0), (0, 5, 0), (0, 0, 5) $. The tangency points are $r_1 = (0, 1.5, 2)$, $r_2 = (1.5, 0, 1.5)$, $r_3 = (1.5, \square, 0)$ The GIF image verifies that the generated ellipsoid is indeed tangent to the four planes of the tetrahedron.