My solution approach :-
$|\lfloor x|x|-7\rfloor|<5$
When $x\geq 0$, then;
$|\lfloor x^2-7\rfloor|<5$
$\Rightarrow -5 \lt \lfloor x^2-7\rfloor \lt5$
After this I do know that I'll have to take another case in which $x\lt0$ but I am not able to get how to solve the floor function.
So far, when $x\geq 0$ you got here $$ -5 \lt \lfloor x^2-7\rfloor \lt5 $$ This means $\lfloor x^2-7\rfloor \in \{-4,-3,-2,-1,0,1,2,3,4\}$ or in other words $x^2 - 7 \in [-4,5)$. Therefore we have $$ x^2 \in [3,12), $$ or $$ x\in[\sqrt{3},2\sqrt{3}). $$ Note that $x\in(-2\sqrt{3}, -\sqrt{3}]$ is excluded because we only take $x\geq 0$.
For the case $x\leq 0$, we have $$ -5 < \lfloor -x^2 - 7 \rfloor < 5. $$ But $\lfloor -x^2 - 7 \rfloor \leq -7 < -5$, therefore there are no solutions for the case $x\leq 0$.
To sum it all up, the solution set of the equation $|\lfloor x|x|-7\rfloor|<5$ is $x\in[\sqrt{3},2\sqrt{3})$.