If we have a segment defined by points $P1$ and $P2$ and a sphere defined by origin $P3$ and radius $r$. And the segment is part of a line defined by $P1 + u(P2-P1)$. Then if the segment and the sphere intersect, the intersection point lies between $u >= 0$ and $u <=1$. How do we derive the formula for u? The formula below I tried to derive is incorrect.
$$u = \frac{(P3-P1)(P2-P1)}{(P2-P1)(P2-P1)}$$
Do we need to extend the endpoints of the segment distance $r$ along the direction of the line and set the new endpoints in the equation above?
Calling $s_{12}(\lambda) = \lambda p_1 + (1-\lambda) p_2$ and given the sphere as $\|p-p_3\|^2 = r^2$ we have
$$ \|s_{12}(\lambda)-p_3\|^2 = \lambda^2\|p_1\|^2+2\lambda(1-\lambda)p_1\cdot p_2+(1-\lambda)^2\|p_2\|^2 - 2\lambda p_1\cdot p_3-2(1-\lambda)p_2\cdot p_3+\|p_3\|^2= r^2 $$
solving for $\lambda$. If $\ \ 0\le \lambda\le 1$ the segment intersects the sphere.
NOTE
$$ \lambda=\frac{\|p_2\|^2-p_1p_2+p_1p_3-p_2p_3\pm\sqrt{(\|p_2\|^2-p_1p_2+p_1p_3-p_2p_3)^2-\|p_1-p_2\|^2(\|p_3-p_2\|^2-r^2)}}{\|p_1-p_2\|^2} $$
Attached a plot showing the solution for
$$ \cases{ p_1 = (2,1,1)\\ p_2 = (5,6,4)\\ p_3 = (1,0,0)\\ r=3 } $$
giving $\lambda = \frac{1}{43} \left(54-\sqrt{379}\right)$ and after substitution into $\lambda p_1+(1-\lambda)p_2$...