I have two 3D space lines on the same plane (M-R and N-R), and I have two known point on the individual line (M and N). the angle between two lines is unknown. And now I want to draw a circle that is tangent to both of the lines (on the same plane), and touch two known points. I know the center (o) is where two perpendicular of lines in M and N intersect each other. but I do not know how to calculate perpendicular line in 3D space and in specified plane.
Or something like this image:
On
Assuming we know $M,N,R$ with $\|M-R\| = \|N-R\|$ we have
$$ (M-R)\cdot(N-R) = \|M-R\|^2\cos\alpha,\ \ \|O-R\|\cos\left(\frac{\alpha}{2}\right) = \|M-R\|,\ \ r = \|M-O\| = \|M-R\|\sin\left(\frac{\alpha}{2}\right) $$
and also
$$ O-R = \frac{(M+N)/2-R}{\|(M+N)/2-R\|}\frac{\|M-R\|}{\cos\left(\frac{\alpha}{2}\right)} $$
now making $\hat n = \frac{(M-R)\times(N-R)}{\|(M-R)\times(N-R)\|}$ we can define the circle
$$ p = O + r\left(\hat u\cos\theta+\hat v\sin\theta\right) $$
with $\|\hat u\|=\|\hat v\|=1,\ \ \hat u\cdot\hat v=\hat u\cdot\hat n=\hat v\cdot\hat n=0$
First find the coordinates of the center $O$ and the radius $r$. By symmetry $O$ is on the line through $B$ and $\frac{A+C}{2}$. Call $\vec{w} = \frac{A+C}{2}-B$. So $O = B + t\vec{w}$ for some $t$. We can solve this $t$-value from the condition
$$\vec{BC} \perp \vec{CO}.$$
Then take the orthonormal basis for the plane consisting of $v_1 = \frac{\vec{OC}}{|\vec{OC}|}$ and $v_2 = \frac{\vec{CB}}{|\vec{CB}|}$. Now the circle is parametrized by
$$\gamma(t) = O + r\cos(t) v_1 + r\sin(t) v_2.$$