Let $f:(0,1] \to (-\infty,0]$ be a smooth function which is strictly negative on $(0,1)$ and satisfies $f(1)=0$.
Let $\epsilon \in (0,1)$ and let $x,y:(0,\epsilon) \to (0,1]$ be smooth functions satisfying $x(s)y(s)=s$, $x(s) < y(s)$ for every $s \in (0,\epsilon)$.
Suppose that $f \circ x=- y, f \circ y=- x$, i.e. $$ f\big(x(s)\big)=- y(s),f\big(y(s)\big)=- x(s) \tag{1} \label{1} $$ for every $s \in (0,\epsilon)$.
Is it true that $f(x)=x-1$?
In the case where $f(x)=x-1$, equation \eqref{1} implies $x(s)+y(s)=1$, which together with $x(s)y(s)=s$, implies that $$x(s)=\frac{1-\sqrt{1-4s}}{2}\text,y(s)=\frac{1+\sqrt{1-4s}}{2}\text,$$ so $x,y$ are defined on $\left(0,\frac{1}{4}\right]$.
The motivation for this question comes from this optimization problem.
Comment:
I specifically assumed a strict inequality $x<y$. This is important, since otherwise we may get more solutions: Set $x(s)=y(s)=\sqrt s$ and $f(x)=-x$.
(Although $f(x)=-x$ does not satisfy $f(1)=0$, so it needs to be modified after $s=\epsilon$ in order to satisfy this boundary condition as well).
That's far from true. The following construction gives an uncountable family of counterexamples.
For a given $ \epsilon \in ( 0 , 1 ) $, choose $ \alpha \in \left( 1 , \frac 1 { \sqrt \epsilon } \right) $ and take any smooth $ x : ( 0 , \epsilon ) \to ( 0 , 1 ] $ such that:
Define $ y : ( 0 , \epsilon ) \to ( 0 , 1 ] $ by $ y ( s ) = \frac s { x ( s ) } $. Then we'll have the following:
Now define: $$ \begin {cases} g : ( 0 , a ) \to ( - \infty , 0 ] \\ g ( t ) = - y \left( x ^ { - 1 } ( t ) \right) \end {cases} \qquad \begin {cases} h : ( b , c ) \to ( - \infty , 0 ] \\ h ( t ) = - x \left( y ^ { - 1 } ( t ) \right) \end {cases} $$ Then $ g $ and $ h $ are smooth and strictly increasing functions, for every $ t \in ( 0 , a ) $ we have $ - c < g ( t ) < - b $, and for every $ t \in ( b , c ) $ we have $ - a < h ( t ) < 0 $. Thus we can mutually extend $ g $ and $ h $ to a strictly increasing and smooth $ f : ( 0 , 1 ] \to ( - \infty , 0 ] $ with $ f ( 1 ) = 0 $, by choosing suitable smooth functions filling $ [ a , b ] $ and $ [ c , 1 ] $. Then $ f $ will satisfy all the desired properties. As we are very much free in choosing $ x $ and extending $ g $ and $ h $, we end up constructing an uncountably infinite family of such functions.
EDIT:
I would like to add that in situations like this, where there is a very large family of smooth functions satisfying certain properties, it may be useful to search for analytic functions in that family. In many cases, that may give a unique solution, which you are seeking. Analytic functions are very rigid and that makes them very rare among smooth functions. For example, the above construction is based on continuing a smooth function in a much free manner. That can't be done for analytic functions, as the analytic continuation is unique. It's interesting that the functions $ x $, $ y $ and $ f $ given by yourself are analytic on their domains. At the moment, I don't know whether that's the only analytic possible case or not.