I am doing very basics in maths.Please give a simple and easy explanation for this problem.I have roughly $3$ or $4$ months experience in maths. What i tried is i thought of derivative of $(f(x))^2$ which is given to be less than $0$ meaning that function will be decareasing so i ticked B but my answer is wrong.
On
The condition $$f(x)f'(x) <0$$ for all $x\in \mathbb{R}$ tells you that at any time, either you have $f(x) <0 $ and $f'(x) >0$ or $f(x) >0$ and $f'(x) <0$. This of course does not necessarily tell you whether the derivative is always positive or always negative, which you would need for answer $A$ or answer $B$. Can you take it from here?
On
b is not the right answer because suppose there exists $x$, such that $f(x)<0$. Now $f$ is decreasing would imply $f'(x)\ge0$, and hence $f(x)f'(x)>0$, contrary to the given condition.
Now you were right in deducing that $f^2$, is a decreasing function, as $(f^2)'=2ff'$. However $f^2$ is decreasing implies $|f|$ is decreasing. Note that $x^2$ increases strictly with $|x|$ but not $x$.
On
Take into consideration that the derivative is a Darboux function. From $f(x)f'(x) <0$ it follows that $f'(x) \ne 0, \forall x$ therefore either $f'(x) \gt 0,\forall x$ or $f'(x) \lt 0, \forall x$.
If $f'(x) \gt 0,\forall x$ then $f(x) \lt 0,\forall x$ therefore $f$ is negative and increasing. It follows that $|f|$ is decreasing.
Now if $f'(x) \lt 0,\forall x$ then $f(x) \gt 0,\forall x$ therefore $f$ is positive and decreasing. It follows that $|f|$ is decreasing.
Both cases lead to $|f|$ is decreasing, therefore $D$ is correct and $C$ is false. To prove $A,B$ false you can easyly find counterexamples.
The condition $f(x)f'(x)<0$ just means that whenever $f$ is positive, then $f'$ is negative, and conversely. Note that $f$ can never be zero, so it is either always positive or always negative (hence $f'$ is either always negative or always positive).
(A) and (B) are certainly not true, since one can find the counterexamples $f(x)=e^{-x}$ and $f(x)=-e^{-x}$. The first counterexample also shows that (C) is false, since $|f(x)|=f(x)$ there.
If $f$ is positive, then $f'$ is negative, hence $f$ must be decreasing. If $f$ is negative, then $f'$ is positive, hence $f$ must be increasing. But notice that in this case $|f|=-f$, hence $|f|$ is always decreasing, so (D) is true.