I am not very familiar with operators (as I do not study mathematics) and I have just started a Quantum Mechanics course in a university. However, I am not sure what should be the precise order of mathematical operations behind this operator:
$\hat{A}=(x^2\frac{\mathrm{d} }{\mathrm{d} x})^2$
Does it apply to function like this (1):
$\hat{A}=(x^4\frac{\mathrm{d^2} f(x)}{\mathrm{d} x^2})$
or like this (2):
$\hat{A}=(x^2\frac{\mathrm{d} f(x)}{\mathrm{d} x})^2$
I would be sincerely grateful for the insights what is the correct form of applying these type of operators to function and why, since my "Google" search regarding this question was unsuccessful. Thank you very much beforehand!
It means neither of those things. The "square" here refers to functional composition, which means "do this operation twice." Hence
$$\widehat{A} f = x^2 \frac{d}{dx} \left( x^2 \frac{df}{dx} \right) = 2x^3 \frac{df}{dx} + x^4 \frac{d^2 f}{dx^2}.$$
The operators $x$ and $\frac{d}{dx}$ together generate a noncommutative algebra under addition and composition which satisfies some but not all of the rules of ordinary high-school algebra; most prominently, multiplication (composition) is noncommutative, like matrix multiplication. In fact you can think of these operators as infinite-dimensional matrices, and Heisenberg did so (see matrix mechanics).
So $\left( x^2 \frac{d}{dx} \right)^2 \neq x^4 \frac{d^2}{dx^2}$, which it would be if $x$ and $\frac{d}{dx}$ commuted. See Weyl algebra or, for a physics perspective, canonical commutation relations.