Suppose you have three points $a,b,c \in \mathbb{R}^3$, not collinear. Let $p$ belonging to the triangle formed by $a,b,c$. For $x,y \in \mathbb{R}^3$ we define:
$$ \begin{array}{l} P_{y}(x) = \left\langle x,\frac{y}{\lVert y \rVert} \right\rangle \frac{y}{\lVert y \rVert} \\ O_{y}(x) = x - P_{y}(x) \end{array} $$
Since $p$ belongs to the plane spanned by $a,b,c$ the following equation hold $$ p = a + \alpha(b - a) + \beta(c - a) \Rightarrow p - a = \alpha(b - a) + \beta(c - a) $$ My goal is finding $\alpha,\beta$. I'll do the calculations for $\alpha$, because $\beta$ is similar. I observe that
$$ \langle x,O_{x}(y) \rangle = 0 $$
Therofore in the plane equation performing the dot product with $O_{c-a}(b-a)$ leads to $$ \langle p - a, O_{c-a}(b-a) \rangle = \langle \alpha(b - a) + \beta(c - a), O_{c-a}(b-a) \rangle \Rightarrow \langle p - a, O_{c-a}(b-a) \rangle = \alpha \langle b - a , O_{c-a}(b-a) \rangle \Rightarrow \alpha = \frac{\langle p - a, O_{c-a}(b-a) \rangle}{\langle b - a , O_{c-a}(b-a) \rangle}. $$
With the same math I get $$ \beta = \frac{\langle p - a, O_{b-a}(c-a) \rangle}{\langle c - a , O_{b-a}(c-a) \rangle} $$
Is this correct?
You’ve indeed come up with valid formulas for normalized barycentric coordinates relative to $\triangle{abc}$. To see this, we can use the fact that these coordinates can be interpreted as ratios of areas, i.e., the barycentric coordinates of a point $p$ that’s inside or on the triangle are $$[m_a:m_b:m_c]=\frac1{\operatorname{area}(\triangle{abc})}\left[\operatorname{area}(\triangle{pbc}):\operatorname{area}(\triangle{pca}):\operatorname{area}(\triangle{pab})\right].$$ (This also holds for exterior points, but you must use signed areas and take some care with the order of the vertices.)
Your operator $O_y(x)$ computes the orthogonal rejection of $x$ from $y$, so $O_{c-a}(x)$ is the altitude from $x$ to the side $\overline{ac}$ of the triangle. Thus, $$m_b={\frac12\|O_{c-a}(p-a)\|\,\|c-a\| \over \frac12\|O_{c-a}(b-a)\|\,\|c-a\|}={\|O_{c-a}(p-a)\| \over \|O_{c-a}(b-a)\|}$$ and similarly for $m_c$. On the other hand, $\langle x,O_{c-a}(b-a)\rangle = \|x\|\,\|O_{c-a}(b-a)\|\cos\theta$, but $\|x\|\cos\theta$ is just the length of the orthogonal projection of $x$ onto $O_{c-a}(b-a)$, which is the same as the orthogonal rejection of $x$ from $c-a$, $O_{c-a}(x)$, so $$\langle x,O_{c-a}(b-a)\rangle = \|O_{c-a}(x)\|\,\|O_{c-a}(b-a)\|.$$ With this identity, your expression for $\alpha$ becomes $${\langle p-a, O_{c-a}(b-a) \rangle \over \langle b-a , O_{c-a}(b-a) \rangle} = {\|O_{c-a}(p-a)\|\,\|O_{c-a}(b-a)\| \over \|O_{c-a}(b-a)\|\,\|O_{c-a}(b-a)\|} = {\|O_{c-a}(p-a)\| \over \|O_{c-a}(b-a)\|}.$$
We can also compare your expressions to a direct method of computing triangular barycentric coordinates via a linear transformation. W.l.o.g. we can examine this in $\mathbb R^2$. The normalized barycentric coordinates $m_b$ and $m_c$ of a point $p$ are just the coordinates of $p$ in a coordinate system that has $a$ as its origin and $c-a$ and $b-a$ as its bases. To simplify the calculations, we translate the standard coordinate system so that $a$ is at the origin. We then have $$\begin{bmatrix}m_b\\m_c\end{bmatrix} = \begin{bmatrix}x_b&x_c\\y_b&y_c\end{bmatrix}^{-1}\begin{bmatrix}x_p\\y_p\end{bmatrix} = \begin{bmatrix}{x_p y_c-y_p x_c \over x_b y_c-y_b x_c} \\ -{x_p y_b-y_p x_b \over x_b y_c-y_b x_c} \end{bmatrix}.$$ We also have (after some algebraic manipulation) $$\langle p,O_c(b)\rangle = {(x_p y_c-y_p x_c)(x_b y_c-y_b x_c) \over x_c^2+y_c^2}$$ and $$\langle b,O_c(b)\rangle = {(x_b y_c-y_b x_c)^2 \over x_c^2+y_c^2},$$ hence $$\alpha = {\langle p,O_c(b)\rangle \over \langle b,O_c(b)\rangle} = {x_p y_c-y_p x_c \over x_b y_c-y_b x_c} = m_b.$$ A similar computation verifies that $\beta=m_c$. Note that the above expression for $m_b$ can be written as $${\det{\begin{bmatrix}x_p&x_c\\y_p&y_c\end{bmatrix}} \over \det{\begin{bmatrix}x_b&x_c\\y_b&y_c\end{bmatrix}}},$$ which highlights the connection of barycentric coordinates to areas.
As a third way to verify these formulas, we can compute the projections of $p$ onto the lines $\overline{ab}$ and $\overline{ac}$ parallel to $\overline{ac}$ and $\overline{ab}$, respectively. These projections can be computed by intersecting lines. Again, we work in $\mathbb R^2$ and place the origin at $a$ for simplicity. The projection onto $b$ parallel to $c$ is easily computed using homogeneous coordinates: $$\begin{align} ((x_b,y_b,1)\times(0,0,1))\times((x_p,y_p,1)\times(x_c,y_c,0)) &= (y_b,-x_b,0)\times(-y_c,x_c,x_p y_c-y_p x_c) \\ &= \left(-x_b(x_p y_c-y_p x_c), -y_b(x_p y_c-y_p x_c), x_c y_b-y_c x_b\right). \end{align}$$ Converted back to Cartesian coordinates, this is $$\left({x_b(x_p y_c-y_p x_c) \over x_b y_c-y_b x_c}, {y_b(x_p y_c-y_p x_c) \over x_b y_c-y_b x_c}\right) = {x_p y_c-y_p x_c \over x_b y_c-y_b x_c}b = \alpha b.$$ A similar computation shows that the projection onto $\overline{ac}$ is equal to $\beta c$, as desired.