Let $G \subset GL(n,\mathbb{R})$ be a finite-dimensional matrix Lie group. I am using the inner product $$\langle \xi, \eta \rangle := \frac{1}{2}tr(\xi^T\eta)$$ for $\xi, \eta \in \mathfrak{g}$. And of course the induced Lie bracket is $$[\xi,\eta]=\xi\eta-\eta\xi$$ where $\xi\eta$ is matrix multiplication. This is the common layout for matrix Lie groups laid out in Hall's Lie Groups, Lie Algebras, and Representations.
For $G=SO(3)$, we know that $\langle \xi, [\xi, \eta]\rangle = 0$ every single time. I understand the Lie bracket is a kind of a generalization of the cross product, so I was wondering if this holds for arbitrary real matrix Lie groups with this particular inner product?
Counterexample:
$\xi =\pmatrix{1&0\\1&1}, \eta = \pmatrix{2&-1\\0&1}$, $G= GL_2(\mathbb R)$ or any subgroup thereof which contains those elements.
As Callum notes in a comment, if instead you had taken (any scalar multiple of) the usual trace form $\langle x, y \rangle := tr(xy)$, the property would hold.
FYI, if $\mathfrak g$ is any Lie algebra over a field $K$, and $V$ a representation of $\mathfrak g$, there is the important concept of an invariant bilinear form $B: V\times V\rightarrow K$, which is one such that
$$B(Xv,w)+B(v,Xw)=0$$
for all $X \in \mathfrak g$, $v,w \in V$. For the adjoint representation $V= \mathfrak g$, this simplifies to
$$B([v,X], w) = B(v,[X,w])$$
which of course entails $B(X, [X,w])=0$ for all $X,w \in \mathfrak g$. It is well-known that the usual trace form mentioned above is invariant on all of $\mathfrak{gl}_n(K)$. The counterexample shows that your form is not.