Suppose '$f$' is a continuous function from $\mathbb{R}$ to $\mathbb{R}$ and $f(f(a))=a$ for some $a \in \mathbb{R}$ then find the number of solutions of the equation $f(x)=x$.
Options given:
(a) no solution
(b) exactly one solution
(c) at most one solution
(d) atleast three solutions
I tried to solve it like this
$$f(f(a))=a$$ $$f(a)=f^{-1}(a)$$
So the value of function is equal to its inverse at $x=a$.
I tried thinking about that function but it will become a special case.
Is there a mathematically rigorous and simple way to tackle this problem?
You can say there is at least one solution to $f(x)=x$. There may be more, but there is at least one.
Here is a proof. If $f(a)=a$ then $a$ is a solution and we are done. Assume for now that $f(a)<a$.
Consider the function $g(x)=f(x)-x$. Note that $g$ is continuous since $f$ is continuous. Then
$$g(a)=f(a)-a<0$$
and
$$g(f(a))=f(f(a))-f(a)=a-f(a)>0$$
Hence, by the Intermediate Value Theorem for continuous functions, $g(x)$ has a zero between $f(a)$ and $a$. Let's call that zero $c$. Then
$$0=g(c)=f(c)-c$$
so $f(c)=c$ and we have found a solution.
You can repeat this argument if $f(a)>a$. That covers all possibilities regarding $f(a)$ and $a$, so there is at least one solution to $f(x)=x$.
In the edited version of your question, which gives four multiple choice possible answers, none of those choices is correct. An example where there is exactly one solution is $f(x)=-x$, which disproves choices (a) and (d). An example where there are infinitely many solutions is $f(x)=x$. This disproves choices (b) and (c).
Are you sure there was no choice "at least one solution"?