I've got stuck on quite a simple problem and not sure how to proceed.
I have an unknown plane and it contains a point $M(5, 2, 0)$ in it. I also have a point $P(6, 1, -1)$ (distance to the plane is $1$) and $Q(0, 5, 4)$ (distance to the plane is $3$). I tried drawing them as right triangles $MNQ$ and $MPO$, but I couldn't figure out how should I calculate the coordinates of the points, where $Q$ and $P$ are projected on a plane.
Could anyone help me with this? Thanks in advance.
There are two planes which meet your criteria.
I solved the problem with algebra because there are many different arrangements of the points which affect the solution. You can visualise a sphere of radius $1$ around $P$ and a sphere of radius $3$ around $Q$. Your plane has to pass through $M$ and be tangent to both spheres. I think Rory Daulton's suggestion of 4 solutions pictured the case where $M$ can 'see' the gap between the spheres which look like: $~~$ O/O $~~$ O\O $~~$ O_O $~~$ O$\bar{\phantom{x}}$O $~~$ as viewed from $M$ where the O's represent the spheres and the line represents the tangent plane. In your case $M$ is located closer to the line between the spheres.
Let the equation of your plane be $ax+by+cz+d=0$, where $a,b,c$ have been normalised so $\sqrt{a^2+b^2+c^2}=1$. $M$ lies on the plane so we have $5a+2b+0c+d=0$ or $d=-5a-2b$.
The distance from $P(6,1,-1)$ to the plane is $1$, so $$\dfrac{ax_0+by_0+cz_0+d}{\sqrt{a^2+b^2+c^2}} = \dfrac{\left|6a+b-c-5a-2b\right|}{1}=1$$ $$\text{i.e. } \left|a-b-c\right|=1$$ Similarly, the distance from $Q(0,5,4)$ to the plane is 3, so $$\left|0a+5b+4c-5a-2b\right| = 3\\\text{i.e. } \left|-5a+3b+4c\right|=3$$ You can solve these two absolute value equations in a few ways. I turned them into a pair of simultaneous equations $\begin{array}{rrrcl}a&-b&-c&=&r_1\\-5a&+3b&+4c&=&r_2\end{array}$ and looked at the 4 cases where $r_1=\pm1, r_2=\pm3$. Combining the equations gives $-2b-c=5r_1+r2$, or $b=\dfrac{-1}{2}(c+5r_1+r_2)$, and then $a=r_1+b+c = \dfrac{1}{2}(c-3r_1-r_2)$. So now we have $a,b,c$ in terms of $c$ and we can turn the constraint $a^b+b^2+c^2=1$ (remember, we normalised $a,b,c$) into a messy quadratic in $c$: $$3c^2+2r_1c+(17r_1^2+8r_1r_2+r_2^2-2)=0$$ Substituting in the various values, the four cases are:
Ignoring where all of $a,b,c$ are multiplied by -1, and working out $d=-5a-2b$, the two solutions are: