Consider a triangle $pqr$. Its radius edge ratio is defined by $\rho_{pqr} = R_{pqr}/\ell_{\min}$, where $R_{pqr}$ is the triangle's circumradius and $\ell_{\min}$ is the triangle's smallest edge. In what follows, I assume that $pq$ is the smallest edge so that $\ell_{\min} = \|p-q\|$. I am looking for a point $c$ on the perpendicular bisector of $pq$ through the circumcenter, placed such that the radius-edge ratio of $pqc$ is exactly $\beta$ for some given $\beta$. In the case that $\beta \geq 1$, I can show that
$$ c = m + \|p-q\|\left(\beta + \sqrt{\beta^2 - \frac 14}\right)\frac{c_1 - m}{\|m -c_1\|}, $$ where $m = (p + q)/2$ and $c_1$ is circumcenter of $pqr$. This comes from noting that the circumcenter of $pqc$, denoted $c_2$, satisfies $\|c - c_2\| = \beta \|p - q\|$ from the radius-edge ratio requirement, giving $\|p-m\|^2 + \|m-c_2\|^2 = \|p c_2\|^2$ which can be arranged to give the $c$ above.
I am interested in the case $\beta < 1$ especially. Here, the issue is that $\|c-c_2\| = \beta \|p - q\|$ no longer has to hold, since $pq$ may no longer be the shortest edge in this case (since $\beta\|p-q\| < \|p-q\|$). Thus, all I have to start with is $$ \frac{\|c-c_2\|}{\min\{\|p-q\|, \|q-c\|, \|c-p\|\}} = \beta, $$ and $\min\{\|p-q\|, \|q-c\|, \|c-p\|\} = \min\{\|p-q\|, \|p-c\|\}$ since $\|q-c\|=\|c-p\|$. The case for whether $pq$ or $pc$ is the shortest edge depends on the angle $\theta = \angle pqc$. If $\theta \geq 60$, then the shortest edge is $pq$ and so the same formula above applies. Otherwise, if $\theta < 60$, $pc$ is the shortest edge, and so $\|c-c_2\| = \beta \|p - c\|$. It is from here that I am no longer sure how to get a formula for $c$, since $c$ and $c_2$ are coupled in this equation, nor am I sure how to actually get $\theta$ to make this comparison. The only other information I think is available is that $m$, $c$, $c_2$, and $c_1$ are all collinear. The collinearity of $m$, $c$, and $c_2$ can also be written as
$$ \begin{vmatrix} m_x & m_y & 1 \\ c_x & c_y & 1 \\ c_{2x} & c_{2y} & 1 \end{vmatrix} = 0 $$
(from the orient predicate), which is at least one more equation.
Question: How can I use the above information, or any other information, to obtain a formula $c$ for $\beta < 1$ (and $\beta \geq \sqrt 3/3$, since that is the theoretical lower limit of course), distinguishing between the cases $\theta < 60$ and $\theta \geq 60$? Here's an image showing an example of what this situation looks like (where I've just placed $c$ somewhere giving $\theta < 60$, I still don't know how to compute it).
Let $h$ be the distance between $c$ and the edge $pq$. We may scale to make the length of $pq = 1$. The length of the sides $pc$ and $qc$ are then $$\sqrt{\frac 14 + h^2}$$ And if $r$ is the circumradius, then the distance from the circumcenter to $pq$ is $h-r$, and thus $r^2 = \frac 14 + (h-r)^2$, so $1+ 4h^2 - 8hr = 0$. Hence $$r = \frac{4h^2 + 1}{8h}$$
If $h \ge \frac{\sqrt 3}2$, then $pq$ is the shortest side, and $r = \beta$, giving $4h^2 - 8\beta h + 1 = 0$ and $$h = \frac{2\beta + \sqrt{4\beta^2 - 1}}2$$ (the other root is always $<\frac{\sqrt 3}2$ and thus was discarded).
If $h < \frac{\sqrt 3}2$, then the other two sides are shorter than $h$, and so $\beta = \frac r{\sqrt{\frac 14 + h^2}}$ or $$\begin{align}\beta\sqrt{\frac 14 + h^2} &= \frac{4h^2 + 1}{8h}\\ h &= \frac 1{2\sqrt{4\beta^2-1}}\end{align}$$