Let $\mathcal{S}_0:\{Q\in \mathbb{R}^{3\times 3}:\text{tr}Q=0 \hspace{5pt}\text{and}\hspace{5pt} Q_{ij}=Q_{ji} \hspace{5pt} \text{for any $i,j=1,2,3$}\}$ and $$ \widetilde{ \mathcal{E}}(Q) =\int_{\mathbb{R}^3} \dfrac{1}{2}|\nabla Q|^2 +\left(\dfrac{1}{2} |Q|^2 - \sqrt{6} \text{tr} (Q^3) + \dfrac{1}{2} |Q|^4\right)dx. $$ on the space $\mathcal{S}_0$. The Euler Lagrange equation should be $$ -\Delta Q = -Q+3\sqrt{6}(Q^2-\dfrac{1}{3}|Q|^2Id) -2|Q|^2Q.$$
The question is I do not why I cannot put $Q+\varepsilon R$ as a variation and compute the equation, for $R \in \mathcal{S}_0$, and what correct variation should I use? Thank you!
For what it's worth, another way is to use a Lagrange multiplier$^1$ to enforce the traceless condition $${\rm tr}(Q)~=~0.\tag{1}$$ Then the extended functional becomes
$$\begin{align} S[Q,\lambda] ~=~&\int_{\mathbb{R}^3}\! d^3x\left( \frac{1}{2}{\rm tr}( \nabla Q^t\cdot \nabla Q)+ V(|Q|^2) -\sqrt{6}{\rm tr}(Q^3)+\lambda {\rm tr}(Q)\right),\cr |Q|^2~:=~&{\rm tr}(Q^tQ), \cr V(s)~=~&\frac{1}{2}s(1+s),\cr s~\in~\mathbb{R}. \end{align}\tag{2}$$
The Euler-Lagrange (EL) equation reads
$$ 0~=~\frac{\delta S}{\delta Q}~=~-\Delta Q^t + 2V^{\prime}(|Q|^2) Q^t-3\sqrt{6}Q^2 +\lambda \mathbb{1}_{3\times 3}.\tag{3}$$
We can determine the Lagrange multiplier by taking the trace of eq. (3):
$$ 0 ~=~-3\sqrt{6}{\rm tr}(Q^2)+3\lambda.\tag{4} $$
Eqs. (3) & (4) lead to OP's sought-for equation.
$^1$ A similar idea can in principle be used to impose that the matrix $Q=Q^t$ is symmetric.