So I separate this integral from $2$ to $\sqrt5$, from $\sqrt5$ to $\sqrt6, \ldots$ from $3$ to $\pi$. Then I made a substitution of $x^2-1=t$. $dx=\frac{dt}{2 \sqrt{t+1}}$. So the length of every interval is 1.
So I can make the sum. But I have no I idea how to count it. Or I don’t need to change integral to sum? $\{x\}=x-[x]$ or I need to use this?
\begin{align} &\int_2^\pi \{ x^2-1\}\, dx \\ &=\int_2^\pi ( x^2-1) - \lfloor x^2-1\rfloor\, dx \\ &=\int_2^\pi ( x^2-1)\, dx - \int_2^\pi \lfloor x^2-1\rfloor\, dx \\ &=\int_2^\pi ( x^2-1)\, dx -\left( \int_3^\pi \lfloor x^2-1\rfloor\, dx + \sum_{k=4}^8\int_{\sqrt{k}}^{\sqrt{k+1}} \lfloor x^2-1\rfloor\, dx \right)\\ &=\int_2^\pi ( x^2-1)\, dx -\left( \int_3^\pi \lfloor x^2-1\rfloor\, dx + \sum_{k=4}^8\int_{\sqrt{k}}^{\sqrt{k+1}} \lfloor x^2-1\rfloor\, dx \right)\\ &=\int_2^\pi ( x^2-1)\, dx -\left( \int_3^\pi (8)\, dx + \sum_{k=4}^8\int_{\sqrt{k}}^{\sqrt{k+1}} (k-1)\, dx \right)\\ \end{align}
Can you complete this?
Edit:
Let $k$ be a positive integer, then for $\sqrt{k} \le x < \sqrt{k+1}$, we have $k \le x^2 < k+1$ and hence we have $k-1 \le x^2-1 < k$, hence $\lfloor x^2-1\rfloor=k-1$