I understand how to create a traceless symmetric tensor, like $$ \hat{X}_{ij} = X_{ij} - \frac{1}{N}\delta_{ij}X_{hh} $$ with Einstein convention of summing over repeated indices.
(By the way, I'm following here the book "Group Theory in a Nutshell for Physicists", by A. Zee).
But for a 3-tensor I understand that you contract by pairs of indices, but why this normalization factor?
$$ \hat{X}_{ijk} = X_{ijk} - \frac{1}{N + 2}(\delta_{ij}X_{hhk} + \delta_{ik}X_{hhj} + \delta_{jk}X_{hhi}) $$
What is the pattern here? How do you generalize to even higher order tensors? Thanks in advance.
Assuming $X_{ijk}$ is symmetric, by symmetry of notation so is $\hat X_{ijk}$, and thus all traces give the same tensor. Then, the trace of $X$ is $X_{iik}$, and if we trace the quantity in parentheses over $_{ij}$, we get $\delta_{ii} X_{hhk} + \delta_{ik} X_{hhi} + \delta_{ik} X_{hhi}$, but $\delta_{ii} = N$, so the quantity in parentheses is $(N + 2) X_{hhk}$. Thus (after relabeling the dummy index $h$) we see that the coefficient must be $\frac{1}{N + 2}$ for $\hat X$ to be tracefree.
With this in hand, I think you can work out the formula for the tracefree part of a symmetric tensor of general rank for yourself.
Remark We can emphasize the symmetry of the formula by instead writing it as $$\hat X_{ijk} := X_{ijk} - \frac{3}{N + 2} \delta_{(ij} X_{k)hh} .$$ Here, the parentheses enclosing the indices ${}_i, {}_j, {}_k$ indicate that we symmetrize over them.