As the question suggests, I would like to know how to find where a 3D line segment given by two points in 3D space would intersect another 3D object (formed by a collection of vertices, or their respective volume equations).
My goal with this information is to develop a ray-casting algorithm where I determine where on a mesh does the ray-cast hit or doesn't hit (such as on the back side of the mesh).
Here's a picture that helps illustrate my problem: A cube made up of red dots has two rays coming from it. One ray is magenta and starts from a back corner and ends at a yellow dot (so it runs right through the cube). The other ray is white and starts on some point on the front surface of the cube and ends at the yellow dot, this line does not cross through the volume of the cube.
Thank you in advanced for your help!
Decompose your object into a triangular mesh, and inspect the intersection between the ray and the triangle.
Define your ray as passing through origin, in some direction defined by an unit vector $\hat{a}$. (This means that if your ray starts, or passes through some other point, you subtract the coordinates of that point from all triangle coordinates.)
Precalculate the unit normal vector $\hat{n}$ for each triangle beforehand. If the triangle has vertices $\vec{p}_1$, $\vec{p}_2$, and $\vec{p}_3$, then $$\begin{aligned} \vec{n} &= \left(\vec{p}_2-\vec{p}_1\right)\times\left(\vec{p}_3-\vec{p}_1\right) \\ \hat{n} &= \frac{\vec{n}}{\sqrt{\vec{n}\cdot\vec{n}}} \\ \end{aligned}$$ Note that $\hat{n}$ is constant, and does not depend on which point your rays might pass through.
When testing a ray, you first find out the distance $d$ where the ray intersects the plane of the triangle, and the location $\vec{p}$ of that point: $$\begin{aligned} d &= \hat{n} \cdot \vec{p}_1 = \hat{n} \cdot \vec{p}_2 = \hat{n} \cdot \vec{p}_3 \\ \vec{p} &= d\hat{a} \\ \end{aligned}$$
If you are not interested in the exact point where the ray intersects a given triangle, only whether the ray intersects a triangle, precalculate another three constant vectors, edge normal vectors in the plane of the triangle: $$\begin{aligned} \vec{e}_1 &= \hat{n} \times \left( \vec{p}_2 - \vec{p}_1 \right) \\ \vec{e}_2 &= \hat{n} \times \left( \vec{p}_3 - \vec{p}_2 \right) \\ \vec{e}_3 &= \hat{n} \times \left( \vec{p}_1 - \vec{p}_3 \right) \\ \end{aligned}$$ Note that these will point inwards. Then, knowing the point $\vec{p}$ on the plane of the triangle, the point is inside the triangle if and only if $$\left\lbrace\begin{aligned} \vec{e}_1 \cdot \left( \vec{p} - \vec{p}_1 \right) &\ge 0 \\ \vec{e}_2 \cdot \left( \vec{p} - \vec{p}_2 \right) &\ge 0 \\ \vec{e}_3 \cdot \left( \vec{p} - \vec{p}_3 \right) &\ge 0 \\ \end{aligned}\right.$$
If, however, you'd like to know the barycentric coordinates $(u, v)$ within the triangle, use this answer, with $$\begin{aligned} (x_0, y_0, z_0) &= \vec{p}_1 \\ (x_u, y_u, z_u) &= \vec{p}_2 - \vec{p}_1 \\ (x_v, y_v, z_v) &= \vec{p}_3 - \vec{p}_1 \\ \end{aligned}$$ In the $(u, v)$ planar coordinates, the three triangle vertices correspond to $(0,0)$, $(1,0)$, and $(1,1)$, respectively. Point $(u, v)$ is within the triangle if and only if $0 \le u + v \le 1$. The center of the triangle is at $(1/3, 1/3)$.
If you wish to e.g. linearly interpolate between values $c_1$, $c_2$, and $c_3$, use $$c(u, v) = c_1 + u ( c_2 - c_1) + v (c_3 - c_1)$$