Suppose $W$ is a subspace of $M_n(\mathbb{R})$ with property that $\text{trace}(AB) = 0$ for all $A,B \in W$. I want to find the largest possible dimension of $W$.
It seems like the answer is $n(n - 1)/2$, I can find such set of matrices namely the set of all triangular matrices, either upper or lower bot not both, with zero diagonal. They are of the form $$\begin{bmatrix} 0 & * & * & * & \dots & * \\ 0 & 0 & * & * & \dots & *\\ 0 & 0 & 0 & * & \dots & *&\\ \vdots & \vdots & \vdots & \vdots & \ddots & *\\ 0 & 0 & 0 & 0 &\dots & 0\end{bmatrix}$$ (or the lower one).
I have problem showing this is the largest dimension it can get. Clearly one of the basis of this subspace is the set of matrices with entry 1 in one of those $*$ and zero otherwise, there are $n(n - 1)/2$ such matrices.
I can show that when adding one more matrix to this basis the property $\text{trace}(AB)$ can no longer hold. Is this the correct way to prove it? Because I think there's an error with this since I start with the uppertriangular then add one more(not general).
The statement is correct, but the method---showing that your candidate subspace is not a proper subspace of any larger subspace that satisfies the criterion---perhaps requires some additional justification:
Hint Show that $B : (X, Y) \mapsto \operatorname{tr}(X Y)$ is a nondegenerate, symmetric bilinear form in this language, the problem asks for the largest dimension of any totally isotropic subspace $W \leq M(n, \Bbb R)$. Then, apply Witt's Theorem.
Here's an alternative method for showing the claim that doesn't require constructing an explicit candidate, at the cost of using some (well-known) facts about semisimple Lie algebras.
First, we can recognize that $B : (X, Y) \mapsto \operatorname{tr}(X Y)$ coincides with a multiple of the Killing form on the codimension-$1$ subspace $\mathfrak{sl}(n, \Bbb R) := \ker \operatorname{tr}$. But the Lie algebra $\mathfrak{sl}(n, \Bbb R)$ (defined by taking the usual commutator of matrices) is the split real form of a semisimple Lie algebra, so the signature of its Killing form, is equal to the rank of the algebra, namely, $n - 1$, that is, the restriction of $B$ to $\mathfrak{sl}(n, \Bbb R)$ has signature $\left(\frac{1}{2} n (n + 1) - 1, \frac{1}{2} n (n - 1)\right)$.
On the other hand, $M(n, \Bbb R)$ decomposes as the orthogonal direct sum of the span $\operatorname{span}\{1_n\}$ of the identity matrix and $\mathfrak{sl}(n, \Bbb R)$, and $B$ is positive-definite on $\operatorname{span}\{1_n\}$, so $B$ has signature $\left(\frac{1}{2} n (n + 1), \frac{1}{2} n (n - 1)\right)$. Thus, the largest possible dimension of a maximal totally isotropic subspace $W \leq M(n, \Bbb R)$ is $\frac{1}{2} n (n - 1)$.
One more method is to find a pseudo-orthogonal basis for $B$ and use that to compute its signature, though arguably this is no easier than finding a candidate maximal subspace in the first place.