I calculated $\lim_{x\to 0}$ [x]{x} as follows ({x} and [x] mean fractional part function and greatest integer function respectively):- $$\lim_{x\to 0} [x]\{x\}$$ To find the right hand limit, I replaced 'x' by $0+h$ where $h>0$ and ${h\to 0^+}$. Hence we get, $$\lim_{h\to 0^+} [0+h]\{0+h\}$$ As ${h\to 0^+}$ and $h>0$,${0+h\to 0}$ and hence $[0+h]=0$.
And again, as ${h\to 0^+}$, $\{0+h\}=h$. So we get the right hand limit as, $$\lim_{h\to 0^+} (0*h)=0$$
To find the left hand limit I replaced 'x' by $0-h$ where $h>0$ and ${h\to 0^-}$. Hence we get, $$\lim_{h\to 0^-} [0-h]\{0-h\}$$ Now we will get $[-h]=-1$ because $h>0$ and ${h\to 0}$.
Also, $\{-h\}=1-h$. So we get the left hand limit as, $$\lim_{h\to 0} (-1\cdot(1-h))=-1$$
This means RHL and LHL are not equal and hence the limit of the function doesn't exist when ${x\to 0}$.But the correct answer is that it does and is equal to $0$. Where did I go wrong?
What you have done is absolutely correct. Another way to see this is that $\{x\} = x - [x]$ and thus, your limit is simply $$\lim_{h \to 0}\left(h[h] - [h]^2\right).$$ The limit of the first term exists since $h \to 0$ and $[h]$ is bounded.
However, the limit of the second term does not exist as $[h]$ is $-1$ on $(-1, 0)$ whereas it is $0$ on $(0, 1)$.
Thus, the overall limit does not exist.
Here's a graph (courtesy of Desmos):
