From the 1996 Canada National Olympiad. I have emphasised the real point of the question.
Find all real solutions to the following system of equations. Carefully justify your answer.
$\begin{cases} \dfrac{4x^2}{1+4x^2} = y \\ \dfrac{4y^2}{1+4y^2} = z \\ \dfrac{4z^2}{1+4z^2} = x \\ \end{cases}$
Only a single-function is at play:
$f:\mathbb{R}\mapsto \mathbb{R}, f(s) = \dfrac{4s^2}{1+4s^2} \tag{1}$
The system of equations is then:
$f(x)=y, f(y)=z, f(z)=x \tag{2}$
But then repeated iteration gives:
$f^3(x)=x, f^3(y)=y, f^3(z)=z \tag{3}$
We are thus looking for fixed points of the function $f^3$. Now if $x,y,z$ are all fixed points of $f$ they will also be fixed points of $f^3$. We will need to show that $f^3$ has no fixed points other than those of $f$.
To get the fixed points of $f$, solve
$\dfrac{4s^2}{1+4s^2} = s \iff \\ 4s^2 = (1+4s^2)s \iff \\ s(2s^2-1)^2=0$
which has solutions $s=0,\frac{1}{2}$. So, the only solutions so far are $(x,y,z) \in \{(0,0,0),(\frac{1}{2},\frac{1}{2},\frac{1}{2})\}$.
I was guided in what follows by:
To reason about any higher-order fixed points, cycles, etc. first draw the graph of $y = f(x)$. We have
$\begin{align} f'(x) &= \dfrac{8x}{(1+4x^2)^2} \\ f''(x) &= \dfrac{8-96x^2}{(1+4x^2)^3} &= \dfrac{8(1-12x^2)}{(1+4x^2)^3} \end{align}$
Now only the domain $x \in [0,1)$ is of interest as anything outside this range cannot be a fixed point of $f^3$, since the codomain of $f$ is $[0,1)$.
Also
$f(0)=0, f(\frac{1}{2})=\frac{1}{2}, f(1)=\frac{4}{5} \tag{a}$ and $f'(0)=0, f'(\frac{1}{2})=1, f'(1)=\frac{8}{25} \tag{b}$ and $f'(0)=0, f'(x)>0\text{ over }(0,\infty) \tag{c}$ and $f''(\frac{1}{\sqrt{12}})=0, f''(x)>0\text{ over }[0,\frac{1}{\sqrt{12}}), f''(x)<0\text{ over }(\frac{1}{\sqrt{12}},\infty) \tag{d}$
So the graph is as follows:

Should one apply a first-order stability analysis then
- the fixed point at $(0,0)$ is monotone stable
- the fixed point at $(\frac{1}{2},\frac{1}{2})$ is of neutral stability ($y'=1$)
From the concavity of $f$, it appears that $(\frac{1}{2},\frac{1}{2})$ is monotone unstable on the left and monotone stable on the right.
Then the fixed points of $f^3$ are the same, and are of the same nature.
Does it make sense to justify the lack of other fixed points for $f^3$ based on such an analysis, or is it only really applicable in close proximity to the fixed points?
Should I be looking for another approach, e.g. showing $\frac{1}{2} < f(\frac{1}{2}+\epsilon) < \frac{1}{2}+\epsilon,\:\forall\epsilon \in (0,\frac{1}{2})$?
From the system we deduce that: $x, y, z \geq 0$.
Consider the function $f(t) = \dfrac{4t^2}{1+4t^2}, t \geq 0$.
$f'(t) = \dfrac{8t}{(1+4t^2)^2} \geq 0$. Thus $f$ is an increasing function on $[0,\infty)$.
So if WLOG, $x \geq y \geq z \geq 0$, then $f(x) \geq f(y) \geq f(z) \Rightarrow y \geq z \geq x \Rightarrow x \geq y \geq z \geq x \Rightarrow x = y = z \Rightarrow \dfrac{4x^2}{1+4x^2} = x \Rightarrow 4x^2=x+4x^3 \Rightarrow x(4x^2-4x+1) = 0\Rightarrow x(2x-1)^2 = 0 \Rightarrow x = 0,\dfrac{1}{2} \Rightarrow (x,y,z) = (0,0,0), \left(\dfrac{1}{2}, \dfrac{1}{2}, \dfrac{1}{2}\right)$