$$W=\{A\in M_{n\times n}(\mathbb{R})\mid A^T=A, \operatorname{tr}(A)=0\}.$$ We wish to find $W^{\perp}$.
Proving that $W$ is a linear space is easy, but I need the set of matrices $B$ such that $\operatorname{tr}(A^TB)=\operatorname{tr}(AB)=0$. I know that if $A$ is symmetric and $B$ is skew-symmetric then $\operatorname{tr}(AB)=0$, but that is not an if and only if statement. Could someone point me in the right direction? Thanks.
If you want to find a basis for $W^\perp$, start with a basis for $W$. Then we have $B \in W^\perp$ if and only if $\langle B, E \rangle = 0$ for every basis element $E$. Since $W$ has a finite basis, this gives you a finite set of linear equations that determine $W^\perp$. You can use these equations to find a basis for $W^\perp$.
For example, if $n = 2$, then $$ W = \operatorname{span}\left\{ \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}, \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix} \right\} $$
So if $B = (b_{i,j}) \in W^\perp$, we have
$$ b_{1,2} + b_{2,1} = 0 \text{ and } b_{1,1} - b_{2,2} = 0. $$
Therefore,
$$ W^\perp = \operatorname{span}\left\{ \begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix}, \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \right\}. $$
Check this calculation and try to extend it to larger $n$.