The fact that $(dB_t)^2=dt$ is it a convention notation, or it can be proved rigorously ? Or more generaly, if $$dY_t=a(t)dt+b(t)dB_t,$$ does the equatity $$(dY_t)^2=b(t)^2dt$$ can by justify rigorously ?
If yes, how ? because, I'm not so sure how to do it.
The notation $(dB_t)^2 = dt$ is just a convenient [abuse of] notation. Formally, the notation $(dB_t)^2$ is the [forward] differential of the quadratic variation of $B_t.$
All the formal derivation, therefore, must be done in terms of the quadratic variation. In this context, it certainly is possible to rigorously justify $$(dB_t)^2 := d[B]_t = dt \\ (dY_t)^2 := d[Y]_t = b(t)^2dt$$
It's a happy coincidence that for certain stochastic differential equations, the quadratic variation may be determined via squaring the entire SDE and applying some rules on the product of differentials. This happy coincidence is why we use this particular abuse of notation.