How can I find if a 3d point is in/on/outside of tetrahedron defined by 3d coordinates (the point and the tetrahedron)? This is what I found on ethernet:
You now just check if a point $P$ is on the other side of the plane. The normal of each plane is pointing away from the center of the tetrahedron. So you just have to test against $4$ planes. Your plane equation looks like this: $ax+by+cz+d=0$ Just fill in the point values $(x,y,z)$. If the sign of the result is $>0$ the point is of the same side as the normal, result $== 0$, point lies in the plane, and in your case you want the third option: $<0$ means it is on the backside of the plane. If this is fulfilled for all $4$ planes, your point lies inside the tetrahedron.
So if this is correct, I'm struggling to understand the equation $ax+by+cz+d=0$. If I'm correct the $x,y,z$ stand for the point coordinates, what do $a,b,c,d$ stand for ?
Convex polyhedra can be defined as the intersection of the half-spaces formed by their face planes. Tetrahedron is one of these.
A half-space is really just a plane, that splits space into "inside" and "outside", determined by which side the plane normal points to. The equation of a plane (and a half-space) in 3D is $$x_n x + y_n y + z_n z = d$$ or equivalently $x_n x + y_n y + z_n z - d = 0$. Here, $(x_n, y_n, z_n)$ is the plane normal (vector perpendicular to the plane), and $d$ is its signed distance from origin (measured in lengths of the normal vector, or $\sqrt{x_n^2 + y_n^2 + z_n^2}$). If the signed distance is negative, the plane is in the direction opposite to the normal; if positive, in the same direction as the normal. And by "distance from origin", we mean the minimum distance between the plane and the origin.
If we use $\vec{n} = (x_n , y_n, z_n)$ and $\vec{p} = (x, y, z)$, we can write the above using basic vector algebra as $$\vec{n} \cdot \vec{p} = d$$ or equivalently $\vec{n} \cdot \vec{p} - d = 0$.
For example: $$\left\lbrace ~ \begin{aligned} -1 x +0 y +0 z &= 0 \\ 0 x -1 y +0 z &= 0 \\ 0 x +0 y -1 z &= 0 \\ +1 x +1 y +1 z &= 1 \\ \end{aligned} \right. \quad \iff \quad \left\lbrace ~ \begin{aligned} (-1, 0, 0) \cdot \vec{p} &= 0 \\ ( 0,-1, 0) \cdot \vec{p} &= 0 \\ ( 0, 0,-1) \cdot \vec{p} &= 0 \\ ( 1, 1, 1) \cdot \vec{p} &= 1 \\ \end{aligned} \right.$$ define a tetrahedron whose faces are (planes) $x = 0$, $y = 0$, $z = 0$, and $x + y + z = 1$. (Its vertices happen to be $(0, 0, 0)$, $(1, 0, 0)$, $(0, 1, 0)$, and $(0, 0, 1)$, but that's not important, we'll get to vertices further below.)
Let's name the part of the space the normal vectors point to, "outside". Then, a point $(x, y, z)$ is within that tetrahedron, if and only if $$\left\lbrace ~ \begin{aligned} -1 x +0 y +0 z &\le 0 \\ 0 x -1 y +0 z &\le 0 \\ 0 x +0 y -1 z &\le 0 \\ +1 x +1 y +1 z &\le 1 \\ \end{aligned} \right. \quad \iff \quad \left\lbrace ~ \begin{aligned} (-1, 0, 0) \cdot \vec{p} &\le 0 \\ ( 0,-1, 0) \cdot \vec{p} &\le 0 \\ ( 0, 0,-1) \cdot \vec{p} &\le 0 \\ ( 1, 1, 1) \cdot \vec{p} &\le 1 \\ \end{aligned} \right.$$ are all true. If any one of them is false, the point is outside the tetrahedron. The four faces have normal vectors $(-1, 0, 0)$, $(0, -1, 0)$, $(0, 0, -1)$, and $(1, 1, 1)$, with corresponding signed distances $0$, $0$, $0$, and $1$.
Let's assume you have the coordinates for the four vertices of the tetrahedron, $$\begin{aligned} \vec{v}_1 &= (x_1, y_1, z_1) \\ \vec{v}_2 &= (x_2, y_2, z_2) \\ \vec{v}_3 &= (x_3, y_3, z_3) \\ \vec{v}_4 &= (x_4, y_4, z_4) \\ \end{aligned}$$ There is a very simple procedure to find the four planes/half-spaces defining it from the vertices alone, because three points (that are not collinear) suffice to define a plane.
In other words, we'll consider the four following cases: $$\begin{array}{ccc|c} \vec{v}_A & \vec{v}_B & \vec{v}_C & \vec{v}_D \\ \hline \vec{v}_1 & \vec{v}_2 & \vec{v}_3 & \vec{v}_4 \\ \vec{v}_1 & \vec{v}_2 & \vec{v}_4 & \vec{v}_3 \\ \vec{v}_1 & \vec{v}_3 & \vec{v}_4 & \vec{v}_2 \\ \vec{v}_2 & \vec{v}_3 & \vec{v}_4 & \vec{v}_1 \\ \end{array}$$ Each case produces one face normal vector $\vec{n}$ and corresponding signed distance factor $d$, via $$\left\lbrace ~ \begin{aligned} \vec{t} &= \left( \vec{v}_B - \vec{v}_A \right) \times \left( \vec{v}_C - \vec{v}_A \right) \\ \vec{n} &= \begin{cases} \vec{t}, & \text{ if } \vec{t} \cdot \vec{v}_D \lt 0 \\ -\vec{t}, & \text{ otherwise } \\ \end{cases} \\ d &= \vec{n} \cdot \vec{v}_A = \vec{n} \cdot \vec{v}_B = \vec{n} \cdot \vec{v}_C \\ \end{aligned} \right.$$ where $\cdot$ denotes the dot product, and $\times$ cross product, and $\vec{t}$ is just a temporary vector. $\vec{t} = (0, 0, 0)$ if the tetrahedron is degenerate: all vertices on the same plane, same line, or at the same point. Depending on the order in which the vertices are defined, the sign of the normal vector needs to be corrected to make sure it points "outside".
For several reasons, it is often useful to make sure $\vec{n}$ are unit vectors, i.e. $\left\lVert\vec{n}\right\rVert = \sqrt{\vec{n}\cdot\vec{n}} = 1$. Then, use $$\left\lbrace ~ \begin{aligned} \vec{t} &= \left( \vec{v}_B - \vec{v}_A \right) \times \left( \vec{v}_C - \vec{v}_A \right) \\ \vec{n} &= \begin{cases} \displaystyle \frac{\vec{t}}{\left\lVert\vec{t}\right\rVert}, & \text{ if } \vec{t} \cdot \vec{v}_D \lt 0 \\ -\displaystyle \frac{\vec{t}}{\left\lVert\vec{t}\right\rVert}, & \text{ otherwise } \\ \end{cases} \\ d &= \vec{n} \cdot \vec{v}_A = \vec{n} \cdot \vec{v}_B = \vec{n} \cdot \vec{v}_C \\ \end{aligned} \right.$$ but note that you need to check if $\lVert\vec{t}\rVert = 0$ to avoid division-by-zero; it will be zero (or very small with floating-point numbers) if the tetrahedron is degenerate.
Also, you do not need to check the three possible values for $d$, they will all match to within rounding error, so you can just use any one of the three safely.
As an example of why unit normal vectors are useful, $\vec{n}\cdot\vec{p}-d$ tells you how far point $\vec{p}$ is "outside" the face determined by the unit normal $\vec{n}$ and signed distance $d$. If it is positive, the point is "outside", and the value is the actual distance how far. If it is zero, the point is exactly on the face. If it is negative (or zero), the point is "inside", and the magnitude (absolute value) is the distance.
Without normalizing the normal vectors to unit length, those distances are in units of the corresponding normal lengths. Sometimes it does not matter (for example, plain point-in-convex-polyhedron tests), but sometimes it is useful.