I'm trying to understand how to write the Killing form $B: \mathfrak{g} \times \mathfrak{g} \to \mathbb{R}$ defined by $$ B(X,Y) = \mathrm{Tr}(\mathrm{ad}_{X} \circ \mathrm{ad}_{Y}) $$
I'm trying to do this specifically for the Lie algebra $\mathfrak{g} = \mathfrak{so}(n)$, but I'm getting stuck right at the end. I know I $should$ be getting $B(X,Y) = (n-2)\mathrm{Tr}(XY)$ at the end of the day.
In my solution I try to keep things in terms of a general Lie algebra, and then specify to $\mathfrak{so}(n)$ at the end.
Assume $\mathfrak{g}$ is a matrix of size $n \times n$. Since $\mathrm{ad}_{X}(A) = [X,A]$, I'm thinking of the linear transformation $ad_{X}:\mathfrak{g} \to \mathfrak{g}$ as a matrix of size $n^{2} \times n^{2}$. Since $$ \left( \mathrm{ad}_{X} \circ \mathrm{ad}_{Y}\right)(A) = [X,[Y,A]] = XYA - YAX - XAY + AYX $$
If $M$ is my matrix (with components $M_{(j,k)(p,q)}$) corresponding to $\mathrm{ad}_{X} \circ \mathrm{ad}_{Y}$ in the sense that: $$ \left( \mathrm{ad}_{X} \circ \mathrm{ad}_{Y}\right)(A)_{jk} = \sum_{p,q=1}^{n} M_{(j,k)(p,q)} A_{pq} $$
I'm assuming then that $B(X,Y) = \mathrm{Tr}\left( \mathrm{ad}_{X} \circ \mathrm{ad}_{Y} \right) = \sum_{j,k=1}^{n} M_{(j,k),(j,k)}$. So after a little grinding, I find that: $$ \left( \mathrm{ad}_{X} \circ \mathrm{ad}_{Y}\right)(A)_{jk} = \sum_{p,q=1}^{n} \left[ X_{jp}Y_{pq}A_{qk} - Y_{jp}A_{pq}X_{qk} - X_{jp}A_{pq}Y_{qk} + A_{jp}Y_{pq}X_{qk} \right] \\ = \sum_{p,q=1}^{n} \left[ \sum_{\ell=1}^{n} \left( X_{j\ell}Y_{\ell p} \delta_{kq} + Y_{q\ell} X_{\ell k} \delta_{j p} \right) - X_{jp} Y_{qk} - Y_{jp} X_{qk} \right] A_{pq} \\ $$
Where $\delta$ is the kronecker delta. So my matrix components are: $$ M_{(j,k)(p,q)} = \sum_{\ell=1}^{n} \left( X_{j\ell}Y_{\ell p} \delta_{kq} + Y_{q\ell} X_{\ell k} \delta_{j p} \right) - X_{jp} Y_{qk} - Y_{jp} X_{qk} $$
Finally, this tells me that the Killing-form ends up looking like: $$ B(X,Y) = \sum_{j,k=1}^{n} M_{(j,k)(j,k)} \\ = \sum_{j,k=1}^{n} \left[ \sum_{\ell=1}^{n} \left( X_{j\ell}Y_{\ell j} \delta_{kk} + Y_{k\ell} X_{\ell k} \delta_{j j} \right) - X_{jj} Y_{kk} - Y_{jj} X_{kk} \right] \\ = 2n \mathrm{Tr}(XY) - 2 \mathrm{Tr}(X)\mathrm{Tr}(Y) $$
So $B(X,Y) = 2n \mathrm{Tr}(XY) - 2 \mathrm{Tr}(X)\mathrm{Tr}(Y)$, and so far I'm still working with a general Lie algebra $\mathfrak{g}$.
Looking at $\mathfrak{so}(n)$, I know this this is the set of skew-symmetric real matrices of size $n \times n$, which means they are all traceless. This simplifies my answer to $B(X,Y) = 2n \mathrm{Tr}(XY)$.
However, $2n \neq n-2$! Where am I going wrong in my proof? Am I taking too large of a leap in assuming that I can write the matrix M corresponding to $\mathrm{ad}_{X} \circ \mathrm{ad}_{Y}$ so easily?
Short answer : Not all matrices are skew-symmetric, and $\mathfrak{so}(n)$ has dimension $\frac{n^2-n}{2}$ not $n^2$, so you cannot apply directly your computation on the $n^2$-dimensional space of $n\times n$ matrices, you have to adapt them a little bit.
Longer answer : let $E_{ab}$ be the matrix all of those coefficients are zero except the one at the intersection of the $a$-th row and the $b$-th column, which is equal to one (so $E_{ab}=(\delta_{ai}\delta_{bj})_{1\leq i,j \leq n}$). Let $\phi=\mathrm{ad}_{X} \circ \mathrm{ad}_{Y},Z=XY,T=YX$.
The main formula for $\phi$ in your OP can be rewritten as follows :
$$ \phi(E_{jk})=\sum_{p,q} M_{(j,k),(p,q)} E_{pq} \tag{1} $$
Now, a basis for $\mathfrak{so}(n)$ is $\lbrace D_{jk} \rbrace_{j<k}$ where $D_{jk}=E_{jk}-E_{kj}$. We already have $\phi(E_{jk})$ above, so we need to compute $\phi(D_{jk})$.
In the sequel, by "negligible terms" I mean terms who will not contribute in the final computation of the trace of $\phi$.
$$ \begin{array}{lcl} \phi(E_{jk}) &=& M_{(j,k),(j,k)} E_{jk} + M_{(j,k),(k,j)} E_{kj}+ \text{negligible terms} \\ \phi(E_{kj}) &=& M_{(k,j),(j,k)} E_{jk} + M_{(k,j),(k,j)} E_{kj}+ \text{negligible terms} \\ \phi(D_{jk}) &=& (M_{(j,k),(j,k)}-M_{(k,j),(j,k)})D_{jk} + \text{negligible terms} \\ \end{array}\tag{2} $$
Note that by the formula for $M_{(j,k),(p,q)}$ in the OP, we have for any $j,k$, distinct or not :
$$ \begin{array}{lcl} M_{(j,k),(j,k)} &=& Z_{jj}+T_{kk} - X_{jj} Y_{kk} - X_{kk} Y_{jj} \\ M_{(j,k),(k,j)} &=& \delta_{jk}(Z_{jj}+T_{kk})- X_{jk} Y_{kj} - Y_{jk} X_{kj} \\ \end{array}\tag{3} $$
So the Killing form is
$$ \begin{array}{lcl} B(X,Y) &=& \sum_{j< k} \bigg(Z_{jj} + T_{kk} - X_{jj} Y_{kk} - X_{kk} Y_{jj} + X_{jk} Y_{kj} + Y_{jk} X_{kj} \bigg) \\ &=& \frac{1}{2}\sum_{j\neq k} \bigg(Z_{jj} + T_{kk} - X_{jj} Y_{kk} - X_{kk} Y_{jj} + X_{jk} Y_{kj} + Y_{jk} X_{kj} \bigg) \\ &=& \frac{n-1}{2}(\mathrm{Tr}(Z)+\mathrm{Tr}(T))-\frac{1}{2}\sum_{j\neq k} \bigg( X_{kk} Y_{jj}+X_{jj} Y_{kk} - X_{jk} Y_{kj} - Y_{jk} X_{kj} \bigg) \\ &=& (n-1)\mathrm{Tr}(XY)-\frac{1}{2}\sum_{j\neq k} \bigg( X_{kk} Y_{jj}+X_{jj} Y_{kk} - X_{jk} Y_{kj} - Y_{jk} X_{kj} \bigg) \\ &=& (n-1)\mathrm{Tr}(XY)-\frac{1}{2}\sum_{j,k} \bigg( X_{kk} Y_{jj}+X_{jj} Y_{kk} - X_{jk} Y_{kj} - Y_{jk} X_{kj} \bigg) \\ &=& (n-1)\mathrm{Tr}(XY)-\mathrm{Tr}(X)\mathrm{Tr}(Y)-\frac{1}{2}(\mathrm{Tr}(Z)+\mathrm{Tr}(T)) \\ &=& (n-2)\mathrm{Tr}(XY). \end{array} $$