Find the radius of the circle tangent to $3$ other circles $O_1$, $O_2$ and $O_3$ have radius of $a$, $b$ and $c$
The Wikipedia page about the Problem of Apollonius can be found here
I can't provide you an exact image because the construction is too complex. What have I done is draws tons of lines to attempt to use Pythagoras's theorem and trig to solve it.
On
If the given circles are tangent to one another, use Descartes' theorem.
Otherwise you will need more information, namely not just the radii but also the distances between the given circles, and following one of the methods from the article on Apollonian circles is likely your best bet. You should expand your question to indicate what parameters you use to describe the configuration.
Personally I'd follow the representation using Lie geometry, then extract an expression for the radius from that. But I'd expect it to be a fairly huge expression.
On
I recently need to compute something like this and I finally use Cayley-Menger determinant to find the radius.
Let $ABCD$ be an tetrahedron. Let $a,b,c$ be the sides of base triangle $ABC$ and $a_1,b_1,c_1$ be the distances between apex $D$ and vertices $A,B,C$ respectively. It is known that the volume of tetrahedron $V$ can be computed using the CM determinant:
$$288V^2 = \mathcal{CM}(a,b,c,a_1,b_1,c_1) \stackrel{def}{=} \left| \begin{matrix} 0 & 1 & 1 & 1 & 1\\ 1 & 0 & a_1^2 & b_1^2 & c_1^2 \\ 1 & a_1^2 & 0 & c^2 & b^2\\ 1 & b_1^2 & c^2 & 0 & a^2\\ 1 & c_1^2 & b^2 & a^2 & 0 \end{matrix}\right| $$
If you have three circles centered at $A,B,C$ with radii $r_a, r_b, r_c$ and you want to find a circle with center $D$, radius $r$ touching these $3$ circles. then $ABCD$ will form a degenerate tetrahedron with volume $V = 0$. This means the radius $r$ satisfy an equation of the form:
$$f(r) = 0\quad\text{ where }\quad f(r) \stackrel{def}{=} \mathcal{CM}(a,b,c,r_a + r, r_b + r, r_c + r)$$
This looks horrible at a first glance. However, if you expand this out, you will find $f(r)$ is a quadratic polynomial in $r$.
It is a little bit complicated to write down all the coefficients of $f(r)$. In my code, I just implement a function compute the CM determinant. I use the function to compute the values of $f(r)$ at $r = 0,\pm 1$, backout the coefficients of $f(r)$ and finally solve the quadratic equation.
As pointed out in another answer, there are in general $8$ circles touching the $3$ given circles. In a typically configuration where the interior of three circles are disjoint from each other, the absolute values of the two roots of $f(r)$ are the radii of the innermost and outermost Apollonius' circles. The radii of other $6$ circles can be obtained by flipping the signs of $r_a, r_b, r_c$ in the definition of $f(r)$.
Finally, for completeness, the formula for the CM determinant is following horrible mess: $$\mathcal{CM}(a,b,c,a_1,b_1,c_1) = 2 \times \begin{cases} & a^2(a_1^2(b^2+c^2-a^2) - (a_1^2-b_1^2)(a_1^2-c_1^2))\\ + & b^2(b_1^2(c^2+a^2-b^2) - (b_1^2-c_1^2)(b_1^2-a_1^2))\\ + & c^2(c_1^2(a^2+b^2-c^2) - (c_1^2-a_1^2)(c_1^2-b_1^2))\\ - & (abc)^2\\ \end{cases}$$
If you can endure the algebraic slog (a computer algebra system really helps), you can find some structure that simplifies the solution.
Let the given circles have radii $r_A$, $r_B$, $r_C$, and (non-collinear) centers $A:=(x_A,y_A)$, $B:=(x_B,y_B)$, $C:=(x_C,y_C)$; I specifically used $$A = (0,0) \qquad B = (c,0) \qquad C = (b\cos A, b \sin A)$$
As Wikipedia's Problem of Apollonius entry notes, there are eight solutions that depend on whether each circle's tangency with the target circle is external or internal. The entry's coordinate discussion accommodates this by attaching sign variables to the radii; here, we can just assert that the radii $r_A$, $r_B$, $r_C$ are signed values.
Let the target circle have center $P:=(x_P,y_P)$ and radius $r$. We can express $P$ in barycentric coordinates: $$P = \frac{\alpha A+\beta B+\gamma C}{\alpha+\beta+\gamma} \tag{1}$$
Now, we'll follow Wikipedia's lead and consider the equations $$\begin{align} (x_P-x_A)^2 + (y_P-y_A)^2=(r-r_A)^2 \\ (x_P-x_B)^2 + (y_P-y_B)^2=(r-r_B)^2 \\ (x_P-x_C)^2 + (y_P-y_C)^2=(r-r_C)^2 \end{align} \tag{2}$$ Upon subtracting the latter two equations from the first, we eliminate troublesome quadratic terms, leaving a linear homogeneous system of two equations in the three unknowns $\alpha$, $\beta$, $\gamma$. The reader can verify that we can write the solution as $$\begin{align} \alpha &= a^2 b c \cos A - a^2 r_A (r_A - 2 r) + a b \cos C\,r_B (r_B - 2 r) + c a \cos B\,r_C (r_C - 2 r) \\[4pt] \beta &= a b^2 c \cos B - b^2 r_B (r_B - 2 r) + b c \cos A\,r_C (r_C - 2 r) + a b \cos C\,r_A (r_A - 2 r) \\[4pt] \gamma &= a b c^2 \cos C - c^2 r_C (r_C - 2 r) + c a \cos B\,r_A (r_A - 2 r) + b c \cos A\,r_B (r_B - 2 r) \end{align} \tag{3}$$ where $a$, $b$, $c$ are the sides of $\triangle ABC$. Conveniently, $\alpha+\beta+\gamma = 16 |\triangle ABC|^2$, by Heron's Formula.
Substituting $P$ with these $\alpha$, $\beta$, $\gamma$ values back into the first equation of $(1)$, we get this quadratic in $r$:
$$\begin{align} 0 &= 4 r^2 \left(\;-4|\triangle ABC|^2 + \sum_{cyc} \left(a^2 r_A^2 - 2 b c \cos A\,r_B r_C \right) \;\right) \\ &-4r \left(\; \sum_{cyc}\left( a^2 r_A^3 - b c \cos A (a^2 r_A + r_B^2 r_C + r_B r_C^2) \right) \;\right) \\[4pt] &+a^2 b^2 c^2 + \sum_{cyc} \left( a^2 r_A^4 - 2 b c \cos A (a^2 r_A^2 + r_B^2 r_C^2)\right) \end{align} \tag{8}$$
Somewhat surprisingly, the discriminant simplifies nicely: $$\triangle = 64 |\triangle ABC|^2 \left(a^2 - (r_B - r_C)^2\right)\left( b^2 - (r_C - r_A)^2\right) \left(c^2 - (r_A - r_B)^2\right) \tag{9}$$
The last three factors invite defining, say, $d$, $e$, $f$ such that $$d^2 = a^2 - (r_B - r_C)^2 \qquad e^2 = b^2 - (r_C - r_A)^2 \qquad f^2 = c^2 - (r_A - r_B)^2 \tag{10}$$ Since the differences could be negative, we allow that $d$, $e$, $f$ could be imaginary; but $d^2e^2f^2$ must be non-negative (so, $def$ is real). As it turns out, it's helpful to define (possibly-imaginary) "angles" $D$, $E$, $F$ with $$\cos D = \frac{-d^2+e^2+f^2}{2e f} \qquad \cos E = \frac{-e^2+f^2+d^2}{2fd} \qquad \cos F = \frac{-f^2+d^2+e^2}{2de} \tag{11}$$ and (possibly-imaginary) "area" $|\triangle DEF|$ with $$|\triangle DEF|^2 = \frac1{16}(d+e+f)(-d+e+f)(d-e+f)(d+e-f) \tag{12}$$
How helpful? Well, helpful enough that $(8)$ shrinks to $$\begin{align} 0 &= 16 r^2 |\triangle DEF|^2 - 4r d e f ( d r_A\cos D + er_B \cos E + fr_C \cos F) \\[4pt] &-\left(d^2 e^2 f^2 + d^4 r_A^2 + e^4 r_B^2 + f^4 r_C^2 - 2 e^2 f^2 r_B r_C - 2 f^2 d^2 r_C r_A - 2 d^2 e^2 r_A r_B \right) \end{align} \tag{8'}$$
(If we defined possibly imaginary $s_A$, $s_B$, $s_C$ with $s_A^2 = d^2 r_A$, etc, the constant term could be written $(s_A+s_B+s_C)(-s_A+s_B+s_C)(s_A-s_B+s_C)(s_A+s_B-s_C)-d^2 e^2 f^2$, but that isn't particularly helpful going forward.)
Solving $(8')$ yields
Note that, if we were to multiply-through the $def$ factor, and expand the cosines via $(11)$, the first three terms would only involve even powers $d$, $e$, $f$, making for a real (possibly-negative) value. We know that $def$ is real, so the $\pm 2def\,|\triangle ABC|$ term is also real. This guarantees that $r$ itself is real.
Since the signs of $r_A$, $r_B$, $r_C$ account for the eight possible tangency configurations, presumably one choice of "$\pm$" is always extraneous. Can we decide which one ahead of time? Maybe, but I'm running out of steam ...
Substituting $(13)$ into a $def$ version of $(7)$, and dividing-through by a common factor, yields a final-ish form for parameter $\alpha$:
If we define $u := d r_A$, $v := e r_B$, $w := f r_C$, we reduce a bit of clutter in $(13)$ and $(14)$: $$\begin{align} r &= \frac{def}{8|\triangle DEF|^2} \left(\;u \cos D + v \cos E + w \cos F\pm 2|\triangle ABC|\;\right) \\ \alpha &= d \left( \begin{array}{l} \phantom{+} \cos D \left( 4 |\triangle DEF|^2 + u^2 + v^2 + w^2 - 2 v w \cos D - 2 w u \cos E - 2 u v \cos F \right) \\[6pt] \pm 2 |\triangle ABC|\; (u - v \cos F - w \cos E) \end{array} \right) \end{align} \tag{15}$$
As a final note, I'll mention that the values $d$, $e$, $f$ have geometric significance related to the power of certain points with respect to the given circles. It's a little tricky to describe, and I don't (yet) see how it might contribute to the discussion, so I won't bother including it.