How to convert the normal vector of a point in some polygonal surface transformed onto the surface of a triangle in space?

Question

It's complicated to explain what the transformation I'm dealing with is but basically it is a transformation that takes a 3D model and maps it to the surface of one polygon on another model. The issue is that I'd prefer to not have to recompute the normal vectors of the post-transformation model at construction time (which are needed for lighting).

There are actually two versions of the transform. I'm not sure which I will pick so ways to convert normals for each will be needed.

First transform:

Given a point $v = (x,y,z)$ in space and the triangle $(p_1, p_2, p_3)$ in space with point normals $(N_1, N_2, N_3)$ the transformation is $\Omega v = p_1x + p_2y + p_3(1 - x - y) + z(N_1x + N_2y + N_3(1 - x - y))$.

Second transform:

Given a point $v = (x,y,z)$ in space and the triangle $(p_1, p_2, p_3)$ in space with point normals $(N_1, N_2, N_3)$ and triangle normal $N_t$ the transformation is $\Omega v = p_1x + p_2y + p_3(1 - x - y) + z(N_1x + N_2y + N_3(1 - x - y)) \frac {1}{<N_t,N_1x + N_2y + N_3(1 - x - y)>}$.

If $v$ has some normal vector associated with it, then what would it map to geometrically in the case of each transform?

---

Note: $<a,b>$ is the dot product between $a$ and $b$.

Answer 1

Another way to look at OP's problem, is to consider it a transformation from one triangular prism to another, or between triangular prism coordinates $(u, v, w)$ and Cartesian coordinates $(x, y, z)$: Each prism is defined by the base triangle $T_1$ vertices $\vec{p}_1$, $\vec{p}_2$, $\vec{p}_3$, and three corresponding translation vectors $\vec{n}_1$, $\vec{n}_2$, $\vec{n}_3$, so that the other end triangle $T_2$ vertices are $(\vec{p}_1+\vec{n}_1)$, $(\vec{p}_2+\vec{n}_2)$, $(\vec{p}_3+\vec{n}_3)$, as shown in the above illustration: $T_1$ is the bottom triangle, and $T_2$ the top triangle, in both triangular prisms.

The triangular prism coordinate system $(u, v, w)$ is such that $$\begin{array}{ccc|c} u & v & w & \text{3D vector} \\ \hline 0 & 0 & 0 & \vec{p}_3 \\ 1 & 0 & 0 & \vec{p}_1 \\ 0 & 1 & 0 & \vec{p}_2 \\ 0 & 0 & 1 & \vec{p}_3 + \vec{n}_3 \\ 1 & 0 & 1 & \vec{p}_1 + \vec{n}_1 \\ 0 & 1 & 1 & \vec{p}_2 + \vec{n}_2 \\ \end{array} \quad \text{with} \quad \left\lbrace \; \begin{array}{rcl} 0 \le & u & \le 1 \\ 0 \le & v & \le 1 \\ 0 \le & u + v & \le 1 \\ 0 \le & w & \le 1 \\ \end{array}\right.$$ In the illustration, darker red lines are $u = 0$, darker green lines are $v = 0$, and the blue lines are $w = 0$. The gray lines are where $u + v = 1$.

The transformation from triangular prism coordinates $(u, v, w)$ to 3D Cartesian coordinates $\vec{v} = (x, y, z)$ is $$\bbox[#ffffef, 1em]{\Omega(u, v, w) = u \vec{p}_1 + v \vec{p}_2 + (1 - u - v) \vec{p}_3 + w \left ( u \vec{n}_1 + v \vec{n}_2 + (1 - u - v) \vec{n}_3 \right )}$$ which is equivalent to $$\begin{aligned} \Omega(u,v,w) & = (1-w)\Bigr( \vec{p}_3 + u (\vec{p}_1 - \vec{p}_3) + w (\vec{p}_2 - \vec{p}_3 ) \Bigr) \\ & + w\Bigr( (\vec{p}_3 + \vec{n}_3) + u \bigr((\vec{p}_1 + \vec{n}_1) - (\vec{p}-3 + \vec{n}_3)\bigr) + v \bigr((\vec{p}_2 + \vec{n}_2) - (\vec{p}_3 + \vec{n}_3)\bigr) \Bigr) \\ \end{aligned}$$ i.e. $(u, v)$ are barycentric coordinates in a triangle, and $w$ interpolates linearly between $T_1$ and $T_2$.

The one additional vector is $\hat{n}_T$, the unit normal vector of $T_1$: $$\hat{n}_T = \frac{\left(\vec{p}_1 - \vec{p}_3\right)\times\left(\vec{p}_1 - \vec{p}_3\right)}{\left\lVert\left(\vec{p}_1 - \vec{p}_3\right)\times\left(\vec{p}_1 - \vec{p}_3\right)\right\rVert}$$ Its direction depends on the order of the three vertices, but let's assume it points towards $T_2$. (If not, just negate all of its components, or swap $\vec{p}_1$ and $\vec{p}_2$ above to correct the ordering.)

OP noted that they desire $T_1$ and $T_2$ to be parallel, and $T_2$ at distance $L$ from $T_1$. This can be achieved by scaling $\vec{n}_1$, $\vec{n}_2$, and $\vec{n}_3$. If $\vec{N}_1$, $\vec{N}_2$, and $\vec{N}_3$ are the original vectors, then the scaled ones are $$\bbox[#ffffef, 1em]{ \vec{n}_1 = \vec{N}_1 \frac{L}{\hat{n}_T \cdot \vec{N}_1} }, \quad \bbox[#ffffef, 1em]{ \vec{n}_2 = \vec{N}_2 \frac{L}{\hat{n}_T \cdot \vec{N}_2} }, \quad \bbox[#ffffef, 1em]{ \vec{n}_3 = \vec{N}_3 \frac{L}{\hat{n}_T \cdot \vec{N}_3} }$$ because $\hat{n}_T \cdot \vec{N}_i$ yields the the length of $\vec{N}_i$ projected to unit vector $\hat{n}_T$, and $\hat{n}_T$ is perpendicular to triangle $T_1$.

---

If we use helper vectors $$\begin{aligned} \vec{b}_0 = ( x_0 , y_0 , z_0 ) &= \vec{p}_3 \\ \vec{b}_1 = ( x_1 , y_1 , z_1 ) &= \vec{p}_1 - \vec{p}_3 \\ \vec{b}_2 = ( x_2 , y_2 , z_2 ) &= \vec{p}_2 - \vec{p}_3 \\ \vec{b}_3 = ( x_3 , y_3 , z_3 ) &= \vec{n}_3 \\ \vec{b}_4 = ( x_4 , y_4 , z_4 ) &= \vec{n}_1 - \vec{n}_3 \\ \vec{b}_5 = ( x_5 , y_5 , z_5 ) &= \vec{n}_2 - \vec{n}_3 \\ \end{aligned}$$ then $$\bbox[#ffffef, 1em]{ \Omega(u,v,w) = \vec{b}_0 + u \vec{b}_1 + v \vec{b}_2 + w ( \vec{b}_3 + u \vec{b}_4 + v \vec{b}_5 )}$$ i.e. $$\Omega(u,v,w) = \begin{cases} x = x_0 + u x_1 + v x_2 + w ( x_3 + u x_4 + v x_5 ) \\ y = y_0 + u y_1 + v y_2 + w ( y_3 + u y_4 + v y_5 ) \\ z = z_0 + u z_1 + v z_2 + w ( z_3 + u z_4 + v z_5 ) \\ \end{cases}$$ A vector from $(u_1 , v_1 , w_1)$ to $(u_2 , v_2 , w_2)$ is $$\vec{\Delta} = \Omega(u_2 , v_2 , w_2 ) - \Omega(u_1 , v_1 , w_1)$$ i.e. $$\bbox[#ffffef, 1em]{ \vec{\Delta} = (u_2 - u_1) \vec{b}_1 + (v_2 - v_1) \vec{b}_2 + (w_2 - w_1) \vec{b}_3 + (u_2 w_2 - u_1 w_1) \vec{b}_4 + (v_2 w_2 - v_1 w_1) \vec{b}_5 }$$

If we examine the axis directions at $(u, v, w)$, we find that $$\begin{aligned} \vec{e}_u(u,v,w) = \frac{d \big( \Omega(u + d u, v, w) - \Omega(u , v , w) \big )}{d u} &= \vec{b}_1 + w \vec{b}_4 \\ \vec{e}_v(u,v,w) = \frac{d \big( \Omega(u, v + d v, w) - \Omega(u , v , w) \big )}{d v} &= \vec{b}_2 + w \vec{b}_5 \\ \vec{e}_w(u,v,w) = \frac{d \big( \Omega(u, v, w + d w) - \Omega(u , v , w) \big )}{d w} &= \vec{b}_3 + u \vec{b}_4 + v\vec{b}_5 \end{aligned}$$ therefore the instant $u$, $v$, $w$ unit axis vectors at $(u, v, w)$ are $$\bbox[#ffffef, 1em]{ \hat{e}_u (u,v,w) = \frac{\vec{b}_1 + w \vec{b}_4}{\left\lVert\vec{b}_1 + w \vec{b}_4\right\rVert} }, \quad \bbox[#ffffef, 1em]{ \hat{e}_v (u,v,w) = \frac{\vec{b}_2 + w \vec{b}_5}{\left\lVert\vec{b}_2 + w \vec{b}_5\right\rVert} }, \quad \bbox[#ffffef, 1em]{ \hat{e}_w (u,v,w) = \frac{\vec{b}_3 + u \vec{b}_4 + w \vec{b}_5}{\left\lVert \vec{b}_3 + u \vec{b}_4 + w \vec{b}_5 \right\rVert} }$$

---

There are three ways of specifying how a direction $(d u , d v , d w)$ at $(u , v , w)$ is transformed to Cartesian coordinates.

Note that $d u$, $d v$, and $d w$ are just the three variables (and not, say, a multiplication with some variable $d$). It is a common notation for differentials, and I used it here out of familiarity. In practice, it means that when you use $(d u , d v , d w)$ to denote a direction, their "length" ($du^2 + dv^2 + dw^2$) is irrelevant, as long as it is not zero; only the direction of that vector matters.

1. Based on $\vec{\Delta}$ above, i.e. picking a point along the desired direction at some distance, converting both that and the origin to 3D points, and using their difference, normalized to unit length, as the Cartesian unit direction vector:

   $$\Omega(u, v, w ; d u , d v , d w) = \frac{\Omega(u + d u , v + d v , w + d w) - \Omega(u , v , w)}{\bigr\lVert \Omega(u + d u , v + d v , w + d w) - \Omega(u , v , w) \bigr\rVert}$$

   This yields

   $$\bbox[#ffffef, 1em]{\Omega(u, v, w ; d u , d v , d w) = \frac{ d u \, \vec{b}_1 + d v \, \vec{b}_2 + d w \, \vec{b}_3 + (u \, d w + w \, d u + d u \, d w) \vec{b}_4 + (v \, d w + w \, d v + d v \, d w) \vec{b}_5}{\bigr\lVert d u \, \vec{b}_1 + d v \, \vec{b}_2 + d w \, \vec{b}_3 + (u \, d w + w \, d u + d u \, d w) \vec{b}_4 + (v \, d w + w \, d v + d v \, d w) \vec{b}_5 \bigr\rVert}}$$

   This is exactly correct only when the mapping is linear.
    

2. Using $(d u , d v , d w)$ as the weights for the unnormalized direction vectors $\vec{e}_u$, $\vec{e}_v$, $\vec{e}_w$:

   $$\Omega(u, v, w ; d u , d v , d w) = \frac{ d u \, \vec{e}_u + d v \, \vec{e}_v + d w \, \vec{e}_w }{\bigr\lVert d u \, \vec{e}_u + d v \, \vec{e}_v + d w \, \vec{e}_w \bigr\rVert}$$

   This yields

   $$\bbox[#ffffef, 1em]{\Omega(u, v, w ; d u , d v , d w) = \frac{ \vec{b}_1 d u + \vec{b}_2 d v + \vec{b}_3 d w + \vec{b}_4 ( u \, dw + w \, du ) + \vec{b}_5 ( v \, dw + w \, dv ) }{\bigr\lVert \vec{b}_1 d u + \vec{b}_2 d v + \vec{b}_3 d w + \vec{b}_4 ( u \, dw + w \, du ) + \vec{b}_5 ( v \, dw + w \, dv ) \bigr\rVert}}$$

3. Using $(d u , d v , d w)$ as the weights for the normalized direction vectors $\hat{e}_u$, $\hat{e}_v$, $\hat{e}_w$:

   $$\Omega(u, v, w ; d u , d v , d w) = \frac{ d u \, \vec{e}_u + d v \, \vec{e}_v + d w \, \vec{e}_w }{\bigr\lVert d u \, \vec{e}_u + d v \, \vec{e}_v + d w \, \vec{e}_w \bigr\rVert}$$

   This yields

   $$\bbox[#ffffef, 1em]{\Omega(u, v, w ; d u , d v , d w) = \frac{ du \frac{ \vec{b}_1 + w \vec{b}_4 }{\left\lVert \vec{b}_1 + w \vec{b}_4 \right\rVert} + dv \frac{ \vec{b}_2 + w \vec{b}_5 }{\left\lVert \vec{b}_2 + w \vec{b}_5 \right\rVert} + dw \frac{ \vec{b}_3 + u \vec{b}_4 + v \vec{b}{5} }{\left\lVert \vec{b}_3 + u \vec{b}_4 + v \vec{b}{5} \right\rVert} }{\left\lVert du \frac{ \vec{b}_1 + w \vec{b}_4 }{\left\lVert \vec{b}_1 + w \vec{b}_4 \right\rVert} + dv \frac{ \vec{b}_2 + w \vec{b}_5 }{\left\lVert \vec{b}_2 + w \vec{b}_5 \right\rVert} + dw \frac{ \vec{b}_3 + u \vec{b}_4 + v \vec{b}{5} }{\left\lVert \vec{b}_3 + u \vec{b}_4 + v \vec{b}{5} \right\rVert} \right\rVert}}$$

These three are all acceptable definitions. The $du^2 + dv^2 + dw^2 = R^2$ spherical shell centered at $(u , v , w)$ is not spherical in Cartesian coordinates, but some other shape resembling an ellipsoid. The definition of "direction" used here is that we essentially try to map that $du^2 + dv^2 + dw^2 = R^2$ shell into a spherical shell in 3D Cartesian coordinates with as little distortion as possible.

The first one provides the direction from $(u , v , w)$ to $(u + du , v + dv , w + dw)$, and is therefore the most versatile numerically, as it essentially maps the spherical shell $du^2 + dv^2 + dw^2 = R^2$ centered at $(u, v, w)$ to the surface of an unit sphere centered at $\Omega(u , v , w)$. This is optimal for those use cases where the direction is defined as pointing from $(u , v , w)$ to $(u + du , v + dv , w + dw)$.

The second and third provide the direction as viewed from $(u , v , w)$; that is, the direction $(du , dv , dw)$ considered relative to the point $(u , v , w)$. In particular, the second form is optimal for use cases such as reflection, where $(du , dv , dw)$ define the direction of the incoming ray. (It is possible to calculate the reflected direction in Cartesian coordinates directly, if we also know the surface normal at point $(u , v , w)$ in those same coordinates.)

In a sense, the second maps the spherical shell to an ellipsoid-like surface, then scales it back to an unit sphere; the third maps the shell to a spherical shell directly. The two only differ in how they map the distortion.

(If you take a balloon with regular dots or latitude and longitude lines, and then squeeze it, you'll see how in different directions angular differences vary. This is that exact effect here. You could think of the three definitions as analogous to three different balloon materials.)

I would personally use the second for lighting effects, when the direction of the incoming ray is in $(u , v , w)$ coordinates (since the reflection occurs at that point); and the first for e.g. target or camera direction calculations (when the direction points at something).

---

There is an inverse transformation $\Omega^{-1}(x, y, z) = (u , v, w)$ at least for the case when the triangles $T_1$ and $T_2$ are parallel (their respective planes at distance $L$ from each other). However, the solution is rather nasty, so I'll omit it. (I only bothered to solve it for the case where the orientation of the triangular prism was optimal, and even then it uses 15 precalculated constants depending on the 6 vectors defining the prism).

Answer 2

If you understood exactly what OP asked about -- in particular, noticed that $\Omega$ denotes a (nonlinear) transformation rather than a matrix --, skip this section.

We have a vector $\vec{v} = \left [ \begin{matrix} x \\ y \\ z \end{matrix} \right ]$ in some odd coordinate system to transform to ordinary 3D coordinates.

The transform is controlled by three points $\vec{p}_1$, $\vec{p}_2$, and $\vec{p}_3$, with their associated unit vectors $\hat{N}_1$, $\hat{N}_2$, and $\hat{N}_3$, respectively; plus optionally an unit vector $\hat{N}_T$ associated with all three points.

The two possible transforms are $$\mathbf{\Omega}_1(\vec{v}) = x \vec{p}_1 + y \vec{p}_2 + (1 - x - y)\vec{p}_3 + z \left ( x \hat{N}_1 + y \hat{N}_1 + (1 - x - y) \hat{N}_3 \right ) \tag{1}\label{NA1}$$ and $$\mathbf{\Omega}_2(\vec{v}) = x \vec{p}_1 + y \vec{p}_2 + (1 - x - y)\vec{p}_3 + \frac{ z \left ( x \hat{N}_1 + y \hat{N}_1 + (1 - x - y) \hat{N}_3 \right )}{\hat{N}_T \cdot \left ( x \hat{N}_1 + y \hat{N}_1 + (1 - x - y) \hat{N}_3 \right )} \tag{2}\label{NA2}$$ For now, let's concentrate on the first version, $\eqref{NA1}$.

For vectors $\vec{v}$ on the $x y$ plane, i.e. $z = 0$, the transform is $$\mathbf{\Omega}_1\left( \left [ \begin{matrix} x \\ y \\ 0 \end{matrix} \right ] \right) = x \vec{p}_1 + y \vec{p}_2 + (1 - x - y) \vec{p}_3 = \vec{p}_3 + x \left( \vec{p}_1 - \vec{p}_3 \right ) + y \left ( \vec{p}_2 - \vec{p}_3 \right )$$ i.e. it defines a point $(x, y)$ in barycentric coordinates with respect to the triangle (2D simplex) defined by vertices $\vec{p}_3$, $\vec{p}_1$, and $\vec{p}_2$.

Essentially, the three points $\vec{p}_1$, $\vec{p}_2$, and $\vec{p}_3$ define one face, and points $\vec{p}_1+\hat{N}_1$, $\vec{p}_2+\hat{N}_2$, and $\vec{p}_3+\hat{N}_3$ the opposing face of a truncated triangular pyramid, or a prism. $0 \le z \le 1$ chooses the plane continuously between the two faces, and $x$ and $y$ are the barycentric coordinates on the triangle ($0 \le x , y , x + y \le 1$) on that plane.

The difference between $\eqref{NA1}$ and $\eqref{NA2}$ is that in the latter, while the transformed surfaces $z = \text{constant}$ are planar, $x = \text{constant}$ and $y = \text{constant}$ may be curved.

If the transformation was simply $$\mathbf{\Omega}\left(\vec{v}\right) = x \vec{p}_1 + y \vec{p}_2 + (1 - x - y)\vec{p}_3 + z \hat{N}_t$$ OP could use a simple 3×3 transformation matrix followed by a translation (that does not depend on $\vec{v}$. But this transformation is not suitable.

If we expand e.g. $\eqref{NA1}$, we get $$\begin{aligned} \mathbf{\Omega}_1\left(\vec{v}\right) & = \vec{p}_3 + x \left( \vec{p}_1 - \vec{p}_3 \right ) + y \left( \vec{p}_2 - \vec{p}_3 \right ) + z \hat{N}_3 \\ \; & + x z \left( \hat{N}_1 - \hat{N}_3 \right ) + y z \left( \hat{N}_2 - \hat{N}_3 \right ) \\ \end{aligned}$$ and immediately see (from $x z$ and $y z$ terms) that the transform (in these coordinates $(x , y , z)$) is not linear.

The question is, how to convert some direction in the mixed $(x,y,z)$ coordinates to an unit vector.

---

Let's say that our point $\vec{v}$ in mixed coordinates has some direction $\vec{d} = \left [ \begin{matrix} i \\ j \\ k \end{matrix} \right ]$ in the same mixed coordinates associated with it, and we wish to find the corresponding unit vector in ordinary Cartesian 3D coordinates.

If we find the vector from $\mathbf{\Omega}_1\left(\vec{v}\right)$ to $\mathbf{\Omega}_1\left(\vec{v} + \vec{d}\right)$ and normalize it to unit length, we get the corresponding transformed direction as an unit vector. This is because in Cartesian coordinates, the mapping is "sufficiently linear". (Details in the next section.)

$$\bbox[#ffffef]{ \begin{aligned}\vec{w} & = \mathbf{\Omega_1}\left( \vec{v} + \vec{d} \right) - \mathbf{\Omega_1}\left(\vec{v}\right) \\ \; & = i \left ( \vec{p}_1 - \vec{p}_3 \right ) + j \left ( \vec{p}_2 - \vec{p}_3 \right ) + k \hat{N}_3 \\ \; & + \left( z i + i k + k x \right) \left( \hat{N}_1 - \hat{N}_3 \right) \\ \; & + \left( z j + j k + k y \right) \left( \hat{N}_2 - \hat{N}_3 \right) \\ \end{aligned} \tag{3a}\label{NA3a}}$$ The transformed unit direction vector is obtained via scaling to unit length, $$\hat{w} = \frac{\vec{w}}{\left\lVert\vec{w}\right\rVert} = \frac{\vec{w}}{\sqrt{\vec{w}\cdot\vec{w}}} \tag{3b}\label{NA3b}$$ Note that $\vec{w}$ and thus $\hat{w}$ depend on both $\vec{v}$ and $\vec{d}$, so in the general case, you cannot avoid having to recalculate the normals for use in e.g. lighting.

In other words, you do not transform a direction $(i, j, k)$ in mixed coordinates alone, but always a direction $(i, j, k)$ with respect to specific mixed coordinate point $(x, y, z)$. Changing any of the six components can change the resulting Cartesian 3D direction vector.

For the second transformation, that also depends on $\hat{N}_T$, $$\begin{aligned}\vec{w} & \approx \mathbf{\Omega_2}\left( \vec{v} + \vec{d} \right) - \mathbf{\Omega_2}\left(\vec{v}\right) \\ \; & = i \left ( \vec{p}_1 - \vec{p}_3 \right ) + j \left ( \vec{p}_2 - \vec{p}_3 \right ) \\ \; & + \frac{\left(\hat{N}_1-\hat{N}_3\right)(z+k)(x+i) + \left(\hat{N}_2-\hat{N}_3\right)(z+k)(y+j)}{(x+i)\hat{N}_T\cdot\left(\hat{N}_1-\hat{N}_3\right) + (y+j)\hat{N}_T\cdot\left(\hat{N}_2-\hat{N}_3\right) + \hat{N}_T\cdot\hat{N}_3} \\ \; & - \frac{z x\left(\hat{N}_1-\hat{N}_3\right) + z y\left(\hat{N}_2-\hat{N}_3\right)}{ x\hat{N}_T\cdot\left(\hat{N}_1-\hat{N}_3\right) + y\hat{N}_T\cdot\left(\hat{N}_2-\hat{N}_3\right)} \\ \end{aligned} \tag{4a}\label{NA4a}$$ so the final unit vector is again $$\hat{w} \approx \frac{\vec{w}}{\left\lVert\vec{w}\right\rVert} = \frac{\vec{w}}{\sqrt{\vec{w}\cdot\vec{w}}} \tag{4b}\label{NA4b}$$ but because of the possible curvature, the direction is only approximate. This is only exact when $\hat{N}_1$, $\hat{N}_2$, and $\hat{N}_3$ are all parallel to $\hat{N}_3$, or at the limit $\lVert\vec{d}\rVert \to 0$. Numerically, the approximation can be controlled by scaling $\vec{d}$ to a suitable small length, just large enough to keep rounding errors within acceptable limits. For example, $\lVert\vec{d}\rVert \approx 0.001$ should be fine for lighting vectors.

---

To understand what the claim that a mapping from one truncated triangular pyramid to another (as used in the transformation above) is "sufficiently linear", let's rewrite $\eqref{NA1}$ as

$$\begin{aligned} \mathbf{\Omega}_1(\vec{v}) & = x \vec{p}_1 + y \vec{p}_2 + (1 - x - y)\vec{p}_3 + z \left ( x \hat{N}_1 + y \hat{N}_1 + (1 - x - y) \hat{N}_3 \right ) \\ \; & = (1 - z)\left ( x \vec{p}_1 + y \vec{p}_2 + (1 - x - y)\vec{p}_3 \right ) \\ \; & + z \left ( x \left( \vec{p}_1 + \hat{N}_1 \right) + y \left( \vec{p}_2 + \hat{N}_2 \right) + (1 - x - y) \left( \vec{p}_3 + \hat{N}_3 \right) \right )\end{aligned}$$ If we use $$ B(x, y; \vec{p}_1 , \vec{p}_2 , \vec{p}_3 ) = x \vec{p}_1 + y \vec{p}_2 + (1 - x - y ) \vec{p}_3 = \vec{p}_3 + x \left(\vec{p}_1 - \vec{p}_3\right) + y \left(\vec{p}_2 - \vec{p}_3\right)$$ to denote a point in triangle $\vec{p}_1$, $\vec{p}_2$, $\vec{p}_3$ specified by barycentric coordinates $(x , y)$, $0 \le x , y , x + y \le 1$, then $$\mathbf{\Omega}_1(\vec{v}) = (1 - z) B\left(x, y; \vec{p}_1 , \vec{p}_2 , \vec{p}_3 \right) + z B\left(x, y; \vec{p}_1 + \hat{N}_1 , \vec{p}_2 + \hat{N}_2 , \vec{p}_3 + \hat{N}_3\right)$$ This transformation from mixed coordinates $(x, y, z)$ is therefore just linear interpolation within a triangular prism, or a truncated triangular pyramid.

This is linear, if the triangular prism or truncated triangular pyramid has planar faces: if and only if $\vec{p}_1$, $\vec{p}_2$, $\vec{p}_1 + \hat{N}_1$, $\vec{p}_2 + \hat{N}_2$ are coplanar; $\vec{p}_1$, $\vec{p}_3$, $\vec{p}_1 + \hat{N}_1$, $\vec{p}_3 + \hat{N}_3$ are coplanar; and $\vec{p}_2$, $\vec{p}_3$, $\vec{p}_2 + \hat{N}_2$, $\vec{p}_3 + \hat{N}_3$ are coplanar. For polygonal modelling, they should be.

The second transform is $$\mathbf{\Omega}_2\left(\vec{v}\right) = B\left(x, y; \vec{p}_1 , \vec{p}_2 , \vec{p}_3\right) + z \frac{B\left(x, y; \hat{N}_1 , \hat{N}_2 , \hat{N}_3 \right)}{\hat{N}_T \cdot B\left(x, y; \hat{N}_1 , \hat{N}_2 , \hat{N}_3 \right)}$$

OP alluded that $\hat{N}_T$ is perpendicular to the plane formed by the three points $\vec{p}_1$, $\vec{p}_2$, $\vec{p}_3$. That situation is equvalent to the case where the three points' $z$ coordinates are zero, $$\vec{p}_1 = \left[\begin{matrix} x_1 \\ y_1 \\ 0 \end{matrix}\right], \vec{p}_2 = \left[\begin{matrix} x_2 \\ y_2 \\ 0 \end{matrix}\right], \vec{p}_3 = \left[\begin{matrix} x_3 \\ y_3 \\ 0 \end{matrix}\right], \vec{N}_1 = \left[\begin{matrix} X_1 \\ Y_1 \\ Z_1 \end{matrix}\right], \vec{N}_2 = \left[\begin{matrix} X_2 \\ Y_2 \\ Z_2 \end{matrix}\right], \vec{N}_3 = \left[\begin{matrix} X_3 \\ Y_3 \\ Z_3 \end{matrix}\right], \vec{N}_T = \left[\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}\right]$$ The transformed point is then at $$\left[\begin{matrix} x_3 + x ( x_1 - x_3) + y ( x_1 - x_3 ) + \frac{ z }{ Z_3 + x (Z_1 - Z_3) + y (Z_2 - Z_3) }\left( X_3 + x (X_1 - X_3) + y (X_2 - X_3) \right) \\ y_3 + x ( y_1 - y_3) + y ( y_1 - y_3 ) + \frac{ z }{ Z_3 + x (Z_1 - Z_3) + y (Z_2 - Z_3) }\left( Y_3 + x (Y_1 - Y_3) + y (Y_2 - Y_3) \right) \\ z \\ \end{matrix}\right]$$ Note the scaling factor $z / \left( Z_3 + x (Z_1 - Z_3) + y (Z_2 - Z_3) \right)$; this is what causes the nonlinearity. The divisor is essentially the cosine of the angle between the interpolated unit vector and the plane normal, so the nonlinearity depends on the maximum angle between $\hat{N}_T$ and $\hat{N}_1$, $\hat{N}_2$, and $\hat{N}_3$.

For this second transformation, you can obviously develop other definitions of "direction", but I don't really see how any of them would work better for polygonal modeling. The fact that this second transform can transform lines to curves, makes it unsuitable for polygonal modeling, in my opinion.

Therefore, my recommendation is to use the first transformation, which has no such drawbacks, and even the direction vector transformation is well defined in exact form (assuming coplanarity of the vectors as mentioned).