I have a triangle $ABC$ and a point $P$ with barycentric coordinates ($\alpha, \beta, \gamma)$ that I want to reflect about the sides $a,b$ and $c$.
Calculating the general expression for a displacement vector perpendicular to $c$ and then using $|PB|=|P'B|$, I got
$$P'=\left(\alpha+x, \beta-\frac{S_A}{c^2}x, \gamma-\frac{S_B}{c^2}x\right)$$
for the reflection about $c$, where
$$x=\frac{-a^2\left((\beta-1)(-\frac{S_B}{c^2})+\gamma(-\frac{S_A}{c^2})\right)-b^2\left(\gamma+\alpha(-\frac{S_B}{c^2})\right)-c^2\left(\alpha(-\frac{S_A}{c^2})+(\beta-1)\right)}{\frac{S_A}{c^2}+\frac{S_B}{c^2}-\frac{S_A S_B}{c^4}}$$
and
$$S_A=\frac{-a^2+b^2+c^2}{2}, S_B=\frac{a^2-b^2+c^2}{2}$$
is Conway's Notation.
Can anyone confirm this or provide an easier formula? Any help is appreciated.
Let's try some brute force with a special case. Assign Cartesian coordinates $$A := (0,0) \qquad B := (c,0) \qquad C := (b\cos A, b\sin A) \tag{1}$$ and define $P$ with barycentric coordinates $(\alpha, \beta, \gamma)$, so that $$P = \frac{\alpha A + \beta B + \gamma C}{\alpha + \beta + \gamma} = \frac{(\beta c+\gamma b \cos A, \gamma b \sin A )}{\alpha+\beta+\gamma} =: (P_x,P_y) \tag{2}$$ Let $P^\prime = (\alpha^\prime, \beta^\prime, \gamma^\prime)$ be the reflection of $P$ in $\overline{AB}$ (the $x$-axis). Then $P^\prime_x=P_x$ and $P^\prime_y=-P_y$, giving the equations $$\begin{align} (\beta\,c+\gamma\,b \cos A)(\alpha^\prime+\beta^\prime+\gamma^\prime)&=(\beta^\prime\,c+\gamma^\prime\,b\cos A)(\alpha+\beta+\gamma) \\[4pt] \gamma\,(\alpha^\prime+\beta^\prime+\gamma^\prime)&= - \gamma^\prime\,(\alpha+\beta+\gamma) \end{align}\tag{3}$$ Solving the system for, say, $\alpha^\prime$ and $\beta^\prime$ gives $$\begin{align} \alpha^\prime &= -\frac{\gamma^\prime}{\gamma\,c} \left(\;\alpha\,c + 2 \gamma\,(c - b \cos A)\;\right) = -\frac{\gamma^\prime}{\gamma\,c} \left(\;\alpha\,c + 2 \gamma\,a \cos B\;\right) \\[4pt] \beta^\prime &= -\frac{\gamma^\prime}{\gamma\,c}\left(\;\beta\,c + 2 \gamma\,b \cos A\;\right) \end{align}\tag{4}$$ from which we can deduce barycentric coordinates, in a smattering of variants,
As a sanity check:
Any point on the $x$-axis has $\gamma=0$; reflection fixes the such a point, and we see from $(5)$ that, indeed $\alpha^\prime : \beta^\prime : 0 = \alpha:\beta:0$.
The reflection of $C$, which has barycentric coordinates $(0,0,1)$, should have Cartesian coordinates $(b\cos A,-b\sin A)$; from the first form in $(5)$, $$\begin{align} \frac{\alpha^\prime A + \beta^\prime B + \gamma^\prime C}{\alpha^\prime+\beta^\prime+\gamma^\prime} &= \frac{A\cdot 2a\cos B + B\cdot 2b\cos A -C\cdot c}{2a\cos B+2b\cos A-c} \\[8pt] &=\frac{(2bc\cos A-bc \cos A, -bc\sin A)}{2c-c} \\[8pt] &= (b\cos A,-b\sin A) \end{align} \tag{6}$$ as expected.