To find a formula for the sums of square of first n natural no, the following method is applied in generating functionology, we know $$\sum_{n=0}^{N} x^n = \frac{x^{N+1} -1}{x-1}$$ So we apply the $\left\lbrace xD \right\rbrace^2$ operator on it and put $ x = 1$ to get the corresponding formula.
The $\left\lbrace xD \right\rbrace^2$ of right side I found as $$\frac{(N+1)^2x^{N+1}}{x-1} - \frac{(N+1)x^{N+2}}{(x-1)^2} - \frac{(N+2)x^{N+2}-x}{(x-1)^2} + \frac{2x(x^{N+2} -x)}{(x-1)^3}$$ Here after substituting $x = 1$, however the denominator are all $0$ .How to eliminate it to get the correct formula i.e $ \frac{N (N+1) (2N +1)} {6}$
Hint. I would set $$\epsilon:=x-1$$ then, as $\epsilon \to 0$, use the binomial theorem to get $$ x^N=(1+\epsilon)^N=1+N\epsilon+\frac{1}{2} N(N-1)\:\epsilon^2+\frac{1}{6} N(N-1)(N-2)\:\epsilon^3+o(\epsilon^3). \tag1 $$ Inserting $(1)$ in your identity gives the sought result.
In fact, by applying $\left\lbrace (1+\epsilon)D_\epsilon \right\rbrace^2$ to $$ \sum_{n=1}^{N} (1+\epsilon)^n=\frac{(1+\epsilon )^{N+1}-1}{\epsilon }, \quad \epsilon\neq0,\tag2 $$ you get, for $\epsilon\neq0$, $$ \sum_{n=1}^{N} n^2\times (1+\epsilon)^n=-\frac{2+3 \epsilon +\epsilon ^2-(1+\epsilon )^{N+1} \left(2+\epsilon \left(3-2 (N+1)+N^2 \epsilon \right)\right)}{\epsilon ^3}.\tag3 $$ Then inserting $(1)$ into $(3)$ leads to $$ \sum_{n=1}^{N} n^2\times (1+\epsilon)^n=\frac{\frac{N(N+1)(2N +1)}{6}\epsilon^3+o(\epsilon^3)}{\epsilon^3}=\frac{N(N+1)(2N +1)}{6}+o(1) $$ and letting $\epsilon \to 0$ gives the announced result.