Let $g$ be a Lie algebra equipped with an ad-invariant nondegenerate symmetric bilinear form $\langle \cdot{,}\cdot \rangle$, let $V$ be a nondegenerate subspace, and let $V^{\perp}$ be the orthogonal complement of $V$.
Question. Suppose that $[[V, V^{\perp}],V^{\perp}] \subset V$, meaning that $[[v, \eta],\xi] \in V$ for all $v \in V$ and $\eta, \xi \in V^{\perp}$. Does it then follow that $V^{\perp}$ is a Lie subalgebra, i.e., that $[V^{\perp} ,V^{\perp}] \subset V^{\perp}$?
Notice that the converse does hold:
Lemma. If $V^{\perp}$ is closed under the Lie bracket, then $[[V,V^{\perp}],V^{\perp}] \subset V$.
Proof. Assume that $[V^{\perp} ,V^{\perp}] \subset V^{\perp}$. Then, since $\langle \cdot{,}\cdot \rangle$ is ad-invariant, \begin{equation*} \langle [v, \eta ], \xi \rangle = \langle v, [\eta, \xi ] \rangle =0 \quad \text{for all $v \in V$ and $\eta, \xi \in V^{\perp}$}, \end{equation*} which implies $[V, V^{\perp}]\subset V$. It follows that $[S, V^{\perp}] \subset V$ for every subset $S$ of $V$, and so $[[V, V^{\perp}],V^{\perp}] \subset V$.
No. Let $\mathfrak{g}=\mathfrak{sl}(n,k)$ with its killing form $B(X,Y) = 2n\mathrm{trace}(XY)$. Let $V = \mathfrak{so}(n,k)$, the skew-symmetric matrices. You can check that $V^\perp$ is the traceless symmetric matrices. In this case $$[V,V] \subset V, \quad [V,V^\perp] \subset V^\perp, \quad [V^\perp,V^\perp] \subset V.$$ It follows easily that $[[V,V^\perp],V^\perp] \subset V$, but $V^\perp$ is not a subalgebra.
The lemma that you have mentioned follows from the fact that in your setting a subalgebra preserves its complement.