Let $f(x)=a(x-2)(x-b)$, where $a,b\in R$ and $a\ne0$. Also, $f(f(x))=a^3\left(x^2-(2+b)x+2b-\frac2a\right)\left(x^2-(2+b)x+2b-\frac ba\right)$, $a\ne0$ has exactly one real zeroes $5$. Find the minima and maxima of $f(x)$.
$f(f(x))$ is a quartic equation. So, it would have $4$ roots. Its real root should occur in pair because complex roots occur in pair. So, I failed to understand the meaning of 'exactly one real zero' in the question. Does that mean its real roots are equal and they are equal to $5$?
Since $f(x)$ has a zero at $2$, $f(f(2))=f(0)=2ab$. But even RHS of $f(f(x))$ is coming out to be $2ab$ at $x=2$.
Putting $x=5$ in $f(f(x))$ gives an equation in $a^2$ and $b^2$. Don't know what to do with it.
Also, $f(x)=a(x^2-(2+b)x+2b)$
$f'(x)=a(2x-2-b)=0\implies x=\frac{2+b}2$ is the minima or maxima.
How to proceed next? Do the critical points of $f(x)$ tell us anything about $f(f(x))$?
From the given form of $f(f(x))$,
$$\begin{align*} f(f(x)) &= a^3 \left[x^2 - (2+b)x + 2b - \frac 2a\right]\left[x^2 - (2+b)x + 2b - \frac ba\right]\\ &= a^3 \left[(x-2)(x-b) - \frac 2a\right]\left[(x-2)(x-b) - \frac ba\right] \end{align*}$$
$(x-2)(x-b)$ has roots at $x=2$ and $x=b$. If we ignore the $-\frac2a$ and $-\frac ba$, then $f(f(2)) = f(f(b)) = 0$.
But if $\frac2a$ and $\frac ba$ are just negative enough, then these two factors of $f(f(x))$:
$$\left[(x-2)(x-b) - \frac 2a\right]\left[(x-2)(x-b) - \frac ba\right]$$
One of these factors would be strictly positive, and the other factor would have a double zero at the $x$ that minimises $(x-2)(x-b)$. Then overall, the product would have just one distinct real zero.
At this point, the requirement that $f(f(5))=0$ becomes useful. Consider that if $x=5$ minimises $(x-2)(x-b)$, then $x = \frac{2+b}{2} = 5$ and so $b=8$.
Then choose $a$ (which would be negative) so that within the product
$$\left[(x-2)(x-8) - \frac 2a\right]\left[(x-2)(x-8) - \frac 8a\right],$$
the minimum of the first factor is $0$ at $x=5$. The second factor would be strictly positive.
$$\begin{align*} (5-2)(5-8) - \frac 2a &= 0\\ a &= -\frac 29\\ \end{align*}$$
To double check:
$$\begin{align*} f(f(x)) &= a^3\left[(x-2)(x-8) - \frac 2a\right]\left[(x-2)(x-8) - \frac 8a\right]\\ &= a^3 [(x-2)(x-8) + 9][(x-2)(x-8) + 36]\\ &= a^3[(x-5+3)(x-5-3)+9][(x-5+3)(x-5-3)+36]\\ &= a^3 [(x-5)^2 - 3^2+9][(x-5)^2 - 3^2 + 36]\\ &= a^3 (x-5)^2 [(x-5)^2 +27] \end{align*}$$
And back to the question,
$$\begin{align*} f(x) &= -\frac 29(x-2)(x-8)\\ f(5) &= -\frac 29(3)(-3) = 2 \end{align*}$$
and the $2$ is a global maximum. This makes sense because
$$f(f(5)) = f(2) = 0,$$
and $f(x)$ is never equal to $8$ for real $x$, otherwise those real $x$ would add zeroes to $f(f(x)) = f(8)$.