Can anybody help me in how to approach this problem. I expanded the fractional part of $x$ and tried to simplify but nothing is happening it is not coming in any format.
For real number $x$ let $\lfloor x \rfloor$ be the greatest integer less than or equal to $x$, and define $\{x\}=x-\lfloor x \rfloor$ to be the fractional part of $x$.
For example, $\{3\} = 0$ and $\{4.56\}=0.56$.
Define $f(x)=x\{x\}$, and let $N$ be the number of real-valued solutions to the equation $f(f(f(x))) = 17$ for $0\le x\le 2020$. Find the remainder when $N$ is divided by $1000$.
A good start is to think about what the function $f$ looks like. Obviously, it will have something to do with integers, so let's orient ourselves around those.
Firstly, if $x \geq 0$, clearly $f(x) \geq 0$, so we are only concerned with the behaviour of $f$ on non-negative reals.
Consider any non-negative integer $k$. As we increase $x$ from $k$ to just under $k+1$, we note that $f(x)$ is strictly increasing, from $0$ at $x = k$ to just below $k+1$ at $x$ just below $k+1$. More formally, we have that $$ \text{$f\vert_{[k,k+1)}$ is a bijection between the domain $[k, k+1)$ and the range $[0, k+1)$.} $$ The range $[a, b)$ is all real numbers that are $\geq a$ and $< b$. Hopefully a diagram of the function is illustrative of this point. The blue line is $y = x$, and the red curve is $f$.
To approach the problem, we'll define two intermediate variables: $x' = f(x)$ and $x'' = f(x')$.
Now, let's first solve $f(x'') = 17$ for real $x''$. This has a solution for exactly one $x''$ in the range $[17, 18)$, exactly one in the range $[18, 19)$, exactly one in the range $[19, 20)$, ..., and so on. There is exactly one because the function is bijective in each of those ranges.
Solving now $f(x') = x''$, let's first consider the solution for the value of $x''$ in the range $[17, 18)$. This will clearly yield exactly one solution for $x'$ in $[17, 18)$, exactly one in $[18, 19)$, exactly one in $[19, 20)$, etc. Now, for the value of $x''$ in $[18, 19)$. We get exactly one solution for $x'$ in $[18, 19)$; and this solution will necessarily be different to the solution we obtained for the previous value of $x''$, because the same value of $x'$ will surely yield the same value of $f(x')$. Similarly, we get a solution in $[19, 20)$ etc. through infinity, and all of these are different to the solutions we previously obtained. Overall, it is not very difficult to see that we will obtain:
Hopefully you see the pattern here. I'll leave it to you to figure out the solution set for $x$, and then count the solutions which are $\leq 2020$.