Suppose we have an expression and want refer to it with some symbol. Usually one can write something like "$X:= \text{Expression}$" to mean the symbol $X$ is defined to be the expression after the symbol "$:=$".
From my understand, before the definition $X$ is supposed to be just a symbol, after the definition it is something you can work with. I guess it's wrong to work with the symbol at the same time you are defining it. For example, if I read "$\sqrt{X}:= \text{Expression}$", I really would think $\sqrt{X}$ is a symbol, not a square root of something (after all, the symbol $X$ is not defined, only the symbol $\sqrt{X}$).
This can brings confusion when some authors want to rush things and introduce a notation at the same time they make something on the notation. In particular, I got stuck for a while with this definition, from Brownian Motion and Stochastic Calculus - Karatsaz, Shreve.
Just to clarify, they use that equal symbol with a triangle over it for definition (it works just as ":=" works). Clearly $[X]_T^2$ is not a symbol to be defined, but the power of $2$ over the symbol $[X]_T$, which was NOT defined. In fact, the authors expect us to guess they made an operation over a definition at the same time they made the definition. I got stuck a while with this. You may argue it's obvious, but I thought $[X]_T$ was some earlier definition I missed and $[X]_T^2$ was not the power of $2$ of $[X]_T$, but something like "$[X]_T$ of order $2$".
To be honest, the very fact that they introduced $[X]_t^2$ as a definition was the reason why I wasn't able to see this as a power of $2$. I never thought it was possible to introduce a notation and make something on this notation at the same time, so I treated $[X]_T^2$ as single symbol defined by the expression given.
I have two questions: Is wrong my understanding about how definitions are made or the authors of this book really made some kind of abuse? If they really made some kind of abuse, how commom is this pratice? I'm really concerned about this, because if this abuse is commom, I will have to read any definition with doubled care from now.
It appears that the authors define $[\square]^2_\square$ which takes as "input" (the two square boxes) a measurable process (your $X$) and a positive real number ($T$ in your text). As such, nothing like $[\square]_\square$ seems to be defined herre and the reader may even suspect that the $[X]_T$ occurring a line later might be a typo (especially in a context where things like $L^2$-norm occur, which are not the square of some "$L$-norm"); in fact this context suggests the natural interpretation of a possible generalized definition $[X]_T^p\triangleq E\int_0^TX^pd\langle M\rangle_t$ and of course this would in general result in $\sqrt[p]{[X]_T^p}\ne \sqrt{[X]_T^2}$.
I agree with you that what the author wants to define and therefore should have written is $$\tag{2.3} [X]_T\triangleq\sqrt{E\int_0^TX^2d\langle M\rangle_t}$$ Or at least he might have formulated it properly as the implicit definition that it is:
It may be the case that the super-size surd caused some typesetting problems.