I want to test if triangle $A$ is in front or behind triangle $b$ from the perspective of a viewing point $P$.
My current procedure is using the standard form of a plane ($Ax+By+Cy+D=0$) but instead of using any point on the plane, I use the triangle points of the triangle a and the viewing point. Then I compare the signs of the results. If the signs of the results of the standard form using the triangle points are the same as the result of the standard form using the viewing point, then the triangle a is on the same side as the viewing point or automatically infront. If only some of the results of the standard form with the triangle points have the same sign as the result of the standard form using the viewing point, the triangle a intersects the triangle $b$. And the last possible outcome is that all the signs of the result of the standard form using the triangle points are the same but the sign of the result of the standard form using the viewing point meaning that triangle $a$ is behind triangle $b$.
But for some reason that doesn't work.
Here is exactly what I did:
- Calculate the normal of the plane of triangle $b$ (using this:https://www.khronos.org/opengl/wiki/Calculating_a_Surface_Normal#Algorithm)
- Normalize
- Calculate Plane constant using a normalized normal vector and point 1 of the triangle as the origin.
- Do the standard for calculations with {$A,B,C$} = normalized normal vector {$X,Y,Z$}, {$x,y,z$} = Triangle A Point $1,2,3$/Viewing Point {$X,Y,Z$} - Triangle $B$ Point $1$ {$X,Y,Z$} and $D$ = Plane constant.
- Compare signs
I'm not sure if this extra Information helps, but I need this for creating a BSP tree.