Given a point P in three-dimensional space, and a line segment from A to B, what is the simplest way to calculate if the projection of P onto AB would lie on the line segment itself (endpoints included)?
I thought of using two Dot products, and then comparing their signs:
Let product1 = (P-A) Dot (B-A)
Let product2 = (P-B) Dot (A-B)
Then return true if either product is zero or if the products have different signs.
Else return false.
This method uses 16 sums and 6 multiplications in all. Is there a simpler way of computing it?
EDIT: I just noticed an error in the example method I posted above. Both products should be using Dot (B-A). So that brings the number of its operations down to 13 sums and 6 multiplications.
Consider the coordinates $(u,v,w)$ of $$N=\vec{AB}$$
The equation of the plane with this normal vector and passing through $P(x_0,y_0,z_0)$ is :
$$f(x,y,z):=ux+vy+wz-p=0 \ \text{with} \ p:=ux_0+vy_0+wz_0.$$
It is well known that the equation of a plane induces two regions : $f(x,y,z):=ux+vy+wz-p \geq 0$ and $f(x,y,z):=ux+vy+wz-p \leq 0$. It is clear that your condition is that $A$ must be in a region and $B$ in the other one.
Said otherwise, it suffices to test whether
$$(ux_A+vy_A+wz_A-p) \times (ux_B+vy_B+wz_B-p) \leq 0$$
In this case $P$ is projected onto line segment $AB$. Otherwise, its projection falls outside this line segment.