I'm not entirely sure if this belongs in Mathematics or GameDev. I'm trying Mathematics first, so please let me know if it's in the wrong place.
In 3D space, I have a plane A given by 3 points A1, A2, and A3, and I need a transform matrix M to scale arbitrary points so their distance to the plane changes by a factor x, along the normal of the plane.
The plane doesn't necessarily pass through the origin so homogeneous coordinates are necessary.
So far I only have convoluted solutions (e.g. rotate and translate plane to match xy plane, scale along z, then reverse rotation and translation) and feel something simpler is possible.
A normal to the plane can be calculated by $n = (A_2 - A_1)\times(A_3 - A_1)$. The unit normal is then $\hat n = \frac n{\|n\|}$ The distance from the plane to the origin is $d = \hat n \cdot A_1$ (or $A_2$ or $A_3$). Thus the equation for points $v$ in the plane is $$\hat n \cdot v = d$$ And the distance of some arbitrary point $w$ from the plane is given by $\hat n \cdot w - d$. More generally, the projection of $w$ onto the plane is $$\widetilde w = w - (\hat n \cdot w - d)\hat n$$
So if we want $w_x$ to a point on the same normal line from the plane passing through $w$, but at a distance of $x$ times the distance of $w$ from the plane, then it would be given by $\widetilde w + x(\hat n \cdot w - d)\hat n$, or: $$\begin{align}w_x &= w - (\hat n \cdot w - d)\hat n+ x(\hat n \cdot w - d)\hat n\\&= w -(1 - x)(\hat n \cdot w - d)\hat n\end{align}$$