If $(f\circ g) (x)=(\sin\sqrt{x})^2$ and $(g\circ f) (x)=\lvert\sin x\rvert$, find $f$ and $g$.
My progress:
By trial and error I got one such pair $f(x) = \sin^2x$, $g(x)=\sqrt{x}$. But I can’t conclude that this is the only such pair or whether there exist other such pairs.
On
No, the pair is not unique. There are infinite (uncountable) pairs! Consider $f(x)=\sin^2 (x)$ and
$$g_A(x)=\begin{cases} \sqrt{x} &\text{if $x\geq 0$ and $x\not=(k\pi)^2$ with $k\in A$,}\\
0 &\text{if $x=(k\pi)^2$ with $k\in A$,}
\end{cases}$$
where $A$ is any subset of $\mathbb{N}^+$.
Note that if $0\leq x\leq 1$ then $g_A(x)=\sqrt{x}$ and $$(g_A\circ f) (x)=g_A(\sin^2(x))=|\sin x|.$$ Moreover, if $x=(k\pi)^2$ with $k\in A$ then $$(f\circ g_A) (x)=\sin^2(0)=0=\sin^2\sqrt{(k\pi)^2}=\sin^2(\sqrt{x}).$$
A different, more abstract approach:
Note that the given formulas make sense only if $x\ge0$. Note also that the functions $\Phi(x)=\sin^2(\sqrt x\,)$ and $\Psi(x)=\sin x$ are given as conjugate, via the function $\alpha(x)=x^2$, that is, $\Phi=\alpha\circ\Psi\circ\alpha^{-1}$. The functions you found were $f=\alpha\circ\Psi$ and $g=\alpha^{-1}$. Then $f\circ g=\alpha\circ\Psi\circ\alpha^{-1}=\Phi$, and $g\circ f=\alpha^{-1}\circ\alpha\circ\Psi=\Psi$.
Clearly, then, we may replace our $\alpha$ by $\bar\alpha=\alpha\circ\psi$ for any $\psi$ commuting with $\Psi$, for then $\bar f=\bar\alpha\circ\Psi$ and $\bar g={\bar\alpha}^{-1}$ will have $\bar f\circ\bar g=\alpha\circ\psi\circ\Psi\circ\psi^{-1}\circ\alpha^{-1}=\Phi$ and $\bar g\circ\bar f=\psi^{-1}\circ\alpha^{-1}\circ\alpha\circ\psi\circ\Psi=\Psi$.
The functions commuting with $\sin$ are rare, but among them are $\sin$ and $\arcsin$, so we have:
Example 1: $f=\sin^2(\sin(x))$, $g=(\alpha\circ\sin)^{-1}=\arcsin(\sqrt x\,)$.
Example 2: $f=x^2$, $g=\sin(\sqrt x\,)$. This is the answer we all should have seen, but it took me a lot of work.
Of course you can use $\psi=\sin^{\circ n}$, where that means iterating the sine $n$ times, or the $(-n)$-fold iteration of $\arcsin$ if $n$ is negative, giving another, different infinite family of solutions.