Let $B_t$ be a standard Brownian motion and $M_t = B_t^2 -t$. From here we are aware of the identity \begin{align} [M]=[B^2]. \end{align} Now, I want to apply Itô's formula to $B_t^2$ and from that find a representation for $[B^2]$ in terms of a single $ds$-integral to check that $M^2-[B^2]$ is a martingale, as it should.
However, applying Itô's formula \begin{align} f(B_t) = f(B_0) + \int_0^t f'(B_s) dB_s + \frac{1}{2} \int_0^t f''(B_s) ds \end{align} gives that \begin{align} B_t^2 &= B_0^2 + 2 \int_0^t B_s\ dB_s + \frac{1}{2} \int_0^t 2 ds \\ B_t^2 &= 0 + [B_s^2]^{s=t}_{s=0} + [s]^{s=t}_{s=0}\\ B^2_t &= B^2_t +t \end{align} Where is the mistake?
Solution:
By Itô's formula we find that \begin{align} 2 \int_0^t B_s\ dB_s = B_t^2 - t = M_t. \end{align} From the fact that $[M]=[B^2]$ and $I(t) = 2 \int_0^t B_s\ dB_s$ is an Itô-integral. We find that \begin{align} [B^2]=[I]= 2 \int_0^t B_s^2\ ds. \end{align}
As already stated in the comment, we usually do not have the formula $\int x dx = x^2/2$, if we replace the ordinary Lebesgue/Riemannian integral by the Itô integral, which is exactly the statement of Itô's lemma, as we have $df(B_t) = f'(B_t) dB_t + \frac{1}{2} f''(B_t) dt$. So in your case, if you are interested in calculating $f(B_s) dB_s$ for a differentiable function, you need the antiderivative of $f$, and on this antiderivative you can apply Itô's formula. For example, if you want to calculate $\cos(B_t) dB_t$, choose $f(x) = \sin(x)$, then by Itô's formula you have $\sin(B_t) - \sin(B_0) = \int_0^t \cos(B_t) dB_t - \frac{1}{2} \int_0^t \sin(B_t) dt$, or in your case you just need to consider $f(x) = x^2/2$.
On the other hand, if you want to prove that $B_t^2 - t$ is indeed a martingale (and hence the quadratic variation of $B_t$ is $t$), apply Itô's formula to $f(x) = x^2$, so you get $B_t^2 = 2 \int_0^t B_s dB_s + \int_0^t 1 dt$, so $B_t^2 - t = \int_0^t B_s dB_s$, which (in this particular case) is a martingale.