We're given that X is a continous r.v. and has a density function of $f_X(x)$ and we're asked to find the density function of $Y=X-\lfloor X \rfloor$ in terms of $f_X(x)$.
We're also given the hint : to use the expression $P(Y \le x ,\lfloor X \rfloor =i)$ where $i$ is integer.
$$P(Y \le x ,\lfloor X \rfloor =i) = P(X - \lfloor X \rfloor \le x, \lfloor X \rfloor = i)$$
$$=P(X - i \le x, \lfloor X \rfloor = i)$$ $$=P(X \le x + i, \lfloor X \rfloor = i)$$ $$=P(X \le x + i,i \le X \lt i + 1)$$
I tried to expand it but this doesn't lead to anywhere. (I'm glad to see any answer with or without the hint). The follow up question in the book is to specify $f_Y(y)$ if $X \sim exp(\lambda)$. (So I need to find a "general" form first)
Comments: The distribution of $Y$ depends on the distribution of $X,$ so your initial job is to express $f_Y$ in terms of $f_X.$
The top histogram below approximates the distribution of $Y$ originating from $X \sim \mathsf{Exp}(rate=0.1)$ and the bottom histogram imitates the distribution of $Y$ originating from $X \sim \mathsf{Unif}(0,10).$ Each is based on a million observations from the relevant $X.$
R code to make the figure:
The distribution of $Y$ is a mixture of several or many random variables with support $[0,1].$ For these two simple cases all of the 'pieces' are of about the same shape. When I did this for $\mathsf{Norm}(100,10)$ it seems that the pieces are of different shapes but have roughly a uniform distribution when combined.
For the follow-up question, I suppose you are meant to use an exponential distribution for $X$ as I did in my first simulation.