I was thinking about the squeeze theorem here. We can denote the $\left[\frac{1}{|x|}\right] =n$, and then try something like:
$$x^2(1+2+3+...+(n-1)+(n-1)) \leq x^2 \frac{n(n+1)}{2} \leq x^2(1+2+3+...+(n-1)+(n+1))$$
But I don't know what to do with $x^2$. Given that $\left[ \frac{1}{|x|} \right]=n$, how do we proceed to find $x^2$?
Given that $$\left[\frac1{|x|}\right]=n$$ we have $$n\le\frac1{|x|}<n+1$$ Therefore $$\frac1n\ge|x|>\frac1{n+1}$$ and thus, $$\frac1{(n+1)^2}<x^2\le\frac1{n^2}$$
Because you seem worry only about an upper bound of the function, I feel that you expect that the limit is $0$. I think it isn't...