I'm interested in the solvability of the functional equation (which we would call it $(*)$) $$f\circ f=g$$ where $g$ is some functions to be defined.
Certainly, even if $(*)$ has a solution, it would not be expected to be simple. For example:
Example 1.1 Consider $g(x)=0$, it easy to show that $$f_1(x)=0$$ $$f_2(x)=D(x)-1\;\text{where $D(x)$ is the Dirichlet function.}$$ $$f_3(x)=\begin{cases}0\;\text{if $x\le0$.}\\-x^2\;\text{if $x>0$}\end{cases}$$ Are all solutions. In fact, by extending $f_2$ and $f_3$, we can get some general form like $$f_2^*=\begin{cases}0\;\text{if $P(x)$.}\\a\;\text{if $\neg P(x)$.}\end{cases}$$ where $P$ is some proposition such that $P(0)$ is true and $a$ is a constant such that $P(a)$ is true. Or: $$f_3^*=\begin{cases}0\;\text{if $x\in B$.}\\h(x)\;\text{if $x\notin B$}\end{cases}$$ where $0\in B$ and $h:\mathbb R\to B$. Notice that $f_3^*$ implies a sad fact that $f$ could nearly have any local property you want (i.e. differentiable or not, continuous or not, etc.). So, it appears to be not simple to find the general form of $(*)$, but how about the existence problem?
To ensure that our work worth itself, the following example of non existence of the solution of $(*)$ by suitably chosen $g$ is presented below:
Example 1.2 Consider the following functional equation $$f(f(x))=g(x)=x^2-2$$ We are going to prove that it has no solution.
Proof
Consider the following two sets $$\mathcal A:=\{x\mid g(x)=x\}=\{-1,2\}$$ $$\mathcal B:=\{x\mid g(g(x))=x\}=\{-1,2,\frac{-1+\sqrt 5}{2},\frac{-1-\sqrt 5}{2}\}$$ Now, assume that $f$ exists.
To be convenient, let $$a:=-1,b:=2,c:=\frac{-1+\sqrt 5}{2},d:=\frac{-1+\sqrt 5}{2}$$ As $g\circ f=f\circ f\circ f=f\circ g$, $g(f(a))=f(a)$ and $g(f(b))=f(b)$, which implies that $f(a),f(b)\in\mathcal A$. By a similar approach we could show that $f(c),f(d)\in\mathcal B$.
If $f(c)=a$, then $g(c)=f(f(c))=f(a)=a$, but as $g(c)=d$, it is a contradiction. Similarly $f(c)\neq b$. And if $f(c)=c$, then $g(c)=f(f(c))=c\neq d$
So $f(c)=d$. But it turns out that$$c=g(d)=f(f(d))=f(f(f(c)))=f(g(c))=f(d)=f(f(c))=g(c)=d$$, a contradiction. And we are done. $\square$
Remark As we could actually show $g(c)=d$ and $g(d)=c$ without using $g$ explicitly (i.e. By considering $g(g(g(c)))$, we could show that $g(c)\in\mathcal B$, followed by $g(c)\neq a,\neq b,\neq c$, giving $g(c)=d$), for every function $g$ with $\#\mathcal A=2$ and $\#\mathcal B=4$, $(*)$ is unsolvable.
So, is there actually a criterion for $g$ such that we could know the solvability of $(*)$? If no, could we extend the above remark to get a class of functions $g$ such that $(*)$ is unsolvable?
Thanks in advance.
There are some necessary conditions that don't require continuity. For example, if $(a,b)$ is a $2$-cycle of $g$ (i.e. $g(a) = b$ and $g(b) = a$), then $(f(a), f(b)$ is also a $2$-cycle, and thus $g$ can't have an odd number of $2$-cycles.