According to https://www.sciencedirect.com/science/article/abs/pii/S004578250200539X?via%3Dihub
We are interested in finding solutions to Maxwell's equations which propagate along the source-free waveguide. The general ansatz for such fields is given by $$ \begin{aligned} & \mathbf{E}\left(\boldsymbol{x}, x_3, t\right)=\left(\boldsymbol{E}(\boldsymbol{x}), E_3(\boldsymbol{x})\right) e^{\jmath\left(\omega t \mp \beta x_3\right)}, \\ & \mathbf{H}\left(\boldsymbol{x}, x_3, t\right)=\left(\boldsymbol{H}(\boldsymbol{x}), H_3(\boldsymbol{x})\right) e^{\jmath\left(\omega t \mp \beta x_3\right)}, \end{aligned} $$ where $\boldsymbol{x} \in \Omega$ and the $x_3$-axis is along the waveguide. The positive number $\omega$ denotes frequency, and $\beta$ is the constant of propagation. $\boldsymbol{E}$ and $\boldsymbol{H}$ are electric and magnetic field components in the plane of the cross section, and $E_3$ and $H_3$ are electric and magnetic components along the waveguide. With ansatz (2.1), the second order 3D Maxwell equations expressed in terms of electric field $\left(\boldsymbol{E}, E_3\right)$ alone reduce to three 2D equations: $$ \left\{\begin{array}{l} \boldsymbol{\nabla} \times\left(\frac{1}{\mu} \boldsymbol{\nabla} \times \boldsymbol{E}\right)-\omega^2 \epsilon \boldsymbol{E}+\frac{\beta^2}{\mu} \boldsymbol{E}-\frac{\jmath \beta}{\mu} \nabla E_3=0 \\ \boldsymbol{\nabla} \circ\left(\frac{1}{\mu} \nabla E_3\right)+\omega^2 \epsilon E_3+\jmath \beta \nabla \circ\left(\frac{1}{\mu} \boldsymbol{E}\right)=0 \\ \boldsymbol{\nabla} \circ(\epsilon \boldsymbol{E})-\jmath \beta \epsilon E_3=0 \end{array}\right. $$
However I am not able to get $$\nabla \times\left(\frac{1}{\mu} \nabla \times E\right)-\omega^2 \epsilon E+\frac{\beta^2}{\mu} E-\frac{\mathrm{j} \beta}{\mu} \nabla E_3=0$$
Especially the term $\frac{\beta^2}{\mu} E$. My confusion is that if $\boldsymbol{\nabla} \times\left(\frac{1}{\mu} \boldsymbol{\nabla} \times \mathbf{E}\right)= \boldsymbol{\nabla} \times\left(\frac{1}{\mu} \boldsymbol{\nabla} \times E'\right) + \boldsymbol{\nabla} \times\left(\frac{1}{\mu} \boldsymbol{\nabla} \times E_3'\right)$, where $E' = (E_1, E_2, 0)$ and $E'_3 = (0, 0, E_3)$, the term $\frac{\beta^2}{\mu} E$ seems to come from nowhere.
Let's stick with the notation in the article.
Our ansatz: \begin{align*} \mathscr{E}(\mathbf{x},x_3,t)&=(\mathbf{E}(\mathbf{x}),E_3(\mathbf{x}))e^{\jmath(\omega t\mp\beta x_3)}\\ \mathscr{H}(\mathbf{x},x_3,t)&=(\mathbf{H}(\mathbf{x}),H_3(\mathbf{x}))e^{\jmath(\omega t\mp\beta x_3)} \end{align*}
Then second-order Maxwell $$\require{cancel} \nabla\times(\frac1\mu\nabla\times\mathscr{E}) +\frac{\partial^2}{\partial t^2}\epsilon\mathscr{E}=0 $$ gives (at $t=0$, using the identity $\nabla\times(\psi\mathbf{A})=\psi\nabla\times\mathbf{A}+(\nabla\psi)\times\mathbf{A}$) \begin{align*} 0&= \nabla\times(\frac1\mu\nabla\times(\mathbf{E}(\mathbf{x}),E_3(\mathbf{x}))e^{\mp\jmath\beta x_3})-\omega^2\epsilon((\mathbf{E}(\mathbf{x}),E_3(\mathbf{x}))e^{\mp\jmath\beta x_3})\\ &=\nabla\times(\frac1\mu\bigg( e^{\mp\jmath\beta x_3}\nabla\times(\mathbf{E}(\mathbf{x}),E_3(\mathbf{x})) + \underbrace{(\nabla e^{\mp\jmath\beta x_3})}_{\parallel\mathbf{e}_3}\times(\mathbf{E}(\mathbf{x}),\cancel{E_3(\mathbf{x})}) \bigg))\\ &\quad-\omega^2\epsilon((\mathbf{E}(\mathbf{x}),E_3(\mathbf{x}))e^{\mp\jmath\beta x_3})\\ &=\nabla\times(\frac1\mu\bigg( e^{\mp\jmath\beta x_3}\nabla\times(\mathbf{E}(\mathbf{x}),E_3(\mathbf{x})) \mp\jmath\beta e^{\mp\jmath\beta x_3}\mathbf{e}_3\times(\mathbf{E}(\mathbf{x}),0) \bigg))\\ &\quad-\omega^2\epsilon((\mathbf{E}(\mathbf{x}),E_3(\mathbf{x}))e^{\mp\jmath\beta x_3})\\ &= e^{\mp\jmath\beta x_3}\nabla\times(\frac1\mu\nabla\times(\mathbf{E}(\mathbf{x}),E_3(\mathbf{x})))+(\nabla e^{\mp\jmath\beta x_3})\times\frac1\mu \nabla\times(\mathbf{E}(\mathbf{x}),E_3(\mathbf{x})))\\ &\quad \mp\frac{\jmath\beta}{\mu}\bigg( (\nabla e^{\mp\jmath\beta x_3})\times(\mathbf{e}_3\times(\mathbf{E}(\mathbf{x}),0)) + e^{\mp\jmath\beta x_3}\nabla\times(\mathbf{e}_3\times(\mathbf{E}(\mathbf{x}),0)) \bigg)\\ &\quad-\omega^2\epsilon((\mathbf{E}(\mathbf{x}),E_3(\mathbf{x}))e^{\mp\jmath\beta x_3})\\ %%%%% &= e^{\mp\jmath\beta x_3}\nabla\times(\frac1\mu\nabla\times(\mathbf{E}(\mathbf{x}),E_3(\mathbf{x})))\mp\jmath\beta e^{\mp\jmath\beta x_3}\mathbf{e}_3\times\frac1\mu (\nabla\times(\mathbf{E}(\mathbf{x}),E_3(\mathbf{x}))))\\ &\quad \mp\frac{\jmath\beta}{\mu}\bigg( \mp\jmath\beta e^{\mp\jmath\beta x_3}\underbrace{\mathbf{e}_3\times(\mathbf{e}_3\times(\mathbf{E}(\mathbf{x}),0))}_{=-(\mathbf{E}(\mathbf{x}),0)} + e^{\mp\jmath\beta x_3}\nabla\times(\mathbf{e}_3\times(\mathbf{E}(\mathbf{x}),0)) \bigg)\\ &\quad-\omega^2\epsilon((\mathbf{E}(\mathbf{x}),E_3(\mathbf{x}))e^{\mp\jmath\beta x_3}) \end{align*} where $\mathbf{e}_3$ is the unit vector in the $x_3$ direction. Now you can take out the $e^{\mp\jmath\beta x_3}$ and I'll leave the rest of the simplification to you.