I am learning the justification of Sample variance
$${\displaystyle {\begin{aligned} \operatorname {E} [\sigma _{Y}^{2}]&=\operatorname {E} \left[{\frac {1}{n}}\sum _{i=1}^{n}\left(Y_{i}-{\frac {1}{n}}\sum _{j=1}^{n}Y_{j}\right)^{2}\right] \quad (1.1)\\[5pt] &={\frac {1}{n}}\sum _{i=1}^{n}\operatorname {E} \left[Y_{i}^{2}-{\frac {2}{n}}Y_{i}\sum _{j=1}^{n}Y_{j}+{\frac {1}{n^{2}}}\sum _{j=1}^{n}Y_{j}\sum _{k=1}^{n}Y_{k}\right ] \quad (1.2)\\[5pt] &={\frac {1}{n}}\sum _{i=1}^{n}\left[{\frac {n-2}{n}}\operatorname {E} [Y_{i}^{2}]-{\frac {2}{n}}\sum _{j\neq i}\operatorname {E} [Y_{i}Y_{j}]+{\frac {1}{n^{2}}}\sum _{j=1}^{n}\sum _{k\neq j}^{n}\operatorname {E} [Y_{j}Y_{k}]+{\frac {1}{n^{2}}}\sum _{j=1}^{n}\operatorname {E} [Y_{j}^{2}]\right] \quad (1.3)\\ &={\frac {1}{n}}\sum _{i=1}^{n}\left[{\frac {n-2}{n}}(\sigma ^{2}+\mu ^{2})-{\frac {2}{n}}(n-1)\mu ^{2}+{\frac {1}{n^{2}}}n(n-1)\mu ^{2}+{\frac {1}{n}}(\sigma ^{2}+\mu ^{2})\right] \quad (1.4)\\ &={\frac {n-1}{n}}\sigma ^{2} \end{aligned}}}$$
i am confused with the usage of square brackets and round brackets.
equation (1.1) is using big round brackets and big square brackets; equation (1.3) is using big square brackets and small(normal) square brackets; equation (1.3) is using big square brackets and small(normal) round brackets.
is there a convention of the usage of square brackets and round brackets?
There is no real difference. Square and round brackets are usually just used to make expressions consisting of large numbers of nested brackets easier to comprehend. Matching the opening and closing brackets by eye is easier when the outer and inner brackets look different. In this case though, the difference is minimal.
Note that many authors like to use square brackets (like $E[x]$) to denote expectation, where a function called $E$ applied to $x$ would usually be written $E(x)$. Not everyone chooses to follow this convention, of course.