$$\left(\frac{d}{{dx}}\right)^2y = \frac{d^2}{{dx}^2}y$$
Can we use both of them to express second order derivative?
I came across this (I am just confused about the notation they have used, so to be sure,I posted this question.)
While searching over internet, I only found $$\left(\frac{d^2y}{{dx}^2}\right)$$ is not equal to $$\left(\frac{dy}{{dx}}\right)^2$$, about which I am pretty clear.
Well, we shouldn't. The symbol ${}^2$ here ""represents"" different operations in some sense. For $\text{d}^2$, it represents composing the operator $\text{d}$ with itself, so $\text{d}^2y=\text{d}(\text{d}y)$, but for ${\text{d}x}^2$ it represents multiplication, it represents $\text{d}x·\text{d}x$.
I say ""represents"" because that's just loose interpretation of the symbol $$\frac{\text{d}}{\text{d}x},$$ it's what inspired that notation and not necessarily what that notation means. You shouldn't really think of Leibniz's notation as a division of differentials, but as a mere operator that takes in a derivable function of one variable and gives back its derivative function.
As long as you understand that squaring there means composing that operator $\text{d}/\text{d}x$ with itself, i.e. $$\left(\frac{\text{d}}{\text{d}x}\right)^2=\frac{\text{d}}{\text{d}x}\left(\frac{\text{d}}{\text{d}x}\right)$$ you can use that notation, but be careful not to use it in other contexts (for example, contexts where differentials are treated as separate entities, not just as notation for derivatives).
EDIT: just to be clear (now that I've seen your edit), in that book they are using that notation as a shorthand. They consider the sum of those operators (the partial derivatives with respect to $x,y,z$) $$\left(\frac{\partial}{\partial x}+\frac{\partial}{\partial y}+\frac{\partial}{\partial x}\right)$$ composed with itself, and writing it as that sum squared is intuitive enough if we think of the ""product"" $\partial/\partial x$ times $u$ as applying that operator to $u$, which would yield $\partial u/\partial x$.