We need to find the domain of $$\dfrac{1}{[x]^2-x-2\{x\}}$$
I tried to find where the denominator is becoming $0$, then I will remove those value of real numbers and I will have my domain.
Simplifying the denominator I get $[x]^2 + 2[x]- 3x=0$ Which clearly indicates that $-3x$ will be an integer so I assumed $3x=n$ and there .. I started putting values of $x$ with intervals like $[1,2), [2,3), [-1,0)$, etc. and found out the value of $n$ which provided me the value of $X$ ($3x=n$) then I matched whether the value that I get will lie in the interval I took. By doing this, I find $4$ values of $x$, which are $x= -\dfrac13, 0, 1, \dfrac83$.
I am not sure of the method. Is there a better or less complicated method
Let $$f(x) = \frac{1}{[x]^2 - x - 2\{x\}}$$
As you observed, the domain of $f$ consists of all real values of $x$ except those that make the denominator equal to zero.
By definition, the fractional part of $x$ is defined to be $\{x\} = x - [x]$, where $[x]$ is the greatest integer less than or equal to $x$. Hence, $x$ is not in the domain of $f$ if \begin{align*} [x]^2 - x - 2\{x\} & = 0\\ [x]^2 - x - 2(x - [x]) & = 0\\ [x]^2 - x - 2x + 2[x] & = 0\\ [x]^2 + 2[x] - 3x & = 0\\ [x]^2 + 2[x] & = 3x\\ \frac{[x]^2 + 2[x]}{3} & = x \end{align*}
Since $x = [x] + \{x\}$, $x$ is not in the domain of $f$ if \begin{align*} [x]^2 - x - 2\{x\} & = 0\\ [x]^2 - ([x] + \{x\}) - 2\{x\} & = 0\\ [x]^2 - [x] - 3\{x\} & = 0\\ [x]^2 - [x] & = 3\{x\}\\ \frac{[x]^2 - [x]}{3} & = \{x\} \end{align*} Furthermore, $0 \leq \{x\} < 1$. Hence, \begin{align*} 0 & \leq \frac{[x]^2 - [x]}{3} < 1\\ 0 & \leq [x]^2 - [x] < 3 \end{align*} Hence, $[x]$ must satisfy this system of inequalities.
\begin{align*} [x]^2 - [x] & \geq 0\\ [x]([x] - 1) & \geq 0 \end{align*} which is satisfied when $[x] \leq 0$ or $[x] \geq 1$.
\begin{align*} [x]^2 - [x] & < 3\\ [x]^2 - [x] + \frac{1}{4} & < 3 + \frac{1}{4}\\ \left([x] - \frac{1}{2}\right)^2 & < \frac{13}{4}\\ \left|[x] - \frac{1}{2}\right| & < \frac{\sqrt{13}}{2} \end{align*} Thus, \begin{align*} -\frac{\sqrt{13}}{2} & \leq [x] - \frac{1}{2} < \frac{\sqrt{13}}{2}\\ \frac{1 - \sqrt{13}}{2} & \leq [x] < \frac{1 + \sqrt{13}}{2} \end{align*} Since $[x]$ is an integer and $$-\frac{3}{2} = \frac{1 - 4}{2} <\frac{1 - \sqrt{13}}{2} < \frac{1 - 3}{2} = -1 \leq [x] \leq 2 = \frac{1 + 3}{2} < \frac{1 + \sqrt{13}}{2} < \frac{1 + 4}{2} = \frac{5}{2},$$ the only possible values of $[x]$ are $-1, 0, 1, 2$.
Note that each of these four values satisfy $[x] \leq 0$ or $[x] \geq 1$.
Therefore, the values of $x$ that are not in the domain of $f$ are \begin{align*} x & = \frac{[-1]^2 + 2[-1]}{3} = \frac{1 - 2}{3} = -\frac{1}{3}\\ x & = \frac{[0]^2 + 2[0]}{3} = \frac{0 + 0}{3} = 0\\ x & = \frac{[1]^2 + 2[1]}{3} = \frac{1 + 2}{3} = 1\\ x & = \frac{[2]^2 + 2[2]}{3} = \frac{4 + 4}{3} = \frac{8}{3} \end{align*} as you found.
Hence, the domain of $f$ is $$\text{Dom}(f) = \left\{x \in \mathbb{R} \mid x \neq -\frac{1}{3}, 0, 1, \frac{8}{3}\right\}$$