Let $f:(1,\infty)\to(1,\infty)$. Find $f$ such that $x^{f(y)^x}=y^{f(x)^y}$ for every $x,y>1$.
I tried to take ln twice, then I took $g(x)=\ln(f(x))$ and $h(x)=\frac{g(x)}{x}$ and I got $h(x)-h(y)=\frac{\ln\left(\frac{\ln(x)}{\ln(y)}\right)}{xy}$. I don’t know how to continue.
On
This can be solved by replacing $y$ by $x+h$ and taking the limit, since this holds for every $x,y$.
Take logarithms twice: $$ \ln\ln x+x \ln f(y) = \ln\ln y+y \ln f(x) $$
Rearrange differentials: $$ x{\ln f(y)-\ln f(x) \over y-x} = {\ln\ln y-\ln\ln x \over y-x} + \ln f(x) $$
Divide by $1/x$ terms: $$ {1\over x}{\ln f(y)-\ln f(x) \over y-x}-{1\over x^2}\ln f(x) = {1\over x^2}{\ln\ln y-\ln\ln x \over y-x} $$
Take the limit $y\to x$: $$ \left( {1\over x}{\ln f(x)} \right)' = {1\over x^2}\left({\ln\ln x}\right)' = \frac1{x^3\ln x}. $$
Finally, solving for $f$: $$ f(x) = \exp \biggl( x \int^x \frac1{z^3 \ln z}\,dz +c\biggr) $$ With the Exponential Integral function $\text{Ei}$, a better expression (but harder to justify) is obtained: $$ f(x)=\exp \bigl( x \mathop{\rm Ei}(-2 \ln(x)) \bigr). $$
Note the original expression implies $\lim_{x\to1^+}f(x)=0$, and since $\lim_{x\to1^+}\text{Ei}(x)=-\infty$, then $f(1)=0$ for the previous formulas.
On
I suspect that this function does not exist. Brethlosze has made a great argument and I agree with everything in it, but the answer just does not seem to work. Here is an attempt of showing a continuously differentiable $f$ does not exist. It should also show a $f$ that is continuously differentiable on any interval cannot exist. This would mean $f$ must be quite pathological, if it exists at all.
From the identity (*): $\ln\ln x + x \ln f(y) = \ln\ln y + y \ln f(x)$ for all $x, y \in (1, \infty)$, we can define a bivariate function $F: (1, \infty)^2 \to \mathbb{R}$ by $$F(x, y) = \ln\ln x + x \ln f(y) = \ln\ln y + y \ln f(x).$$ It is twice continuously differentiable: $$\nabla F(x, y) = \pmatrix{\partial_x (\ln\ln x + x \ln f(y)) \\ \partial_y (\ln\ln y + y \ln f(x))} = \pmatrix{\frac{1}{x \ln x} + \ln f(y) \\ \frac{1}{y \ln y} + \ln f(x)},$$ and each component again has continuous partial derivatives. Then, the mixed partial derivatives must be equal: $$\frac{f'(x)}{f(x)} = \partial_x \partial_y F(x, y) = \partial_y \partial_x F(x, y) = \frac{f'(y)}{f(y)}.$$ So we can define $G(x, y) = \frac{f'(x)}{f(x)} = \frac{f'(y)}{f(y)}$. This $G$ is independent of $y$ and of $x$, so it is a constant, say $A$. This implies $f(x) = e^{Ax+B}$ for some constant $B$. However, when you plug this into the identity (*), it fails: $$\text{LHS} = \ln \ln x + x (Ay + B) = \ln \ln y + y (Ax + B) = \text{RHS},$$ simplifying into $$\ln \ln x + B x = \ln \ln y + B y \text{ for all } x, y \in (1, \infty),$$ but no constant $B$ can make this identity hold.
No such $f$ exists. In fact, even extending the codomain to $[0,\infty),$ the only possibility is the constant map $f(x) = 0$.
Firstly, taking logs, the functional equation is $ f(y)^x \log x = f(x)^y \log y,$ and fixing some value of $y,$ observe that as $x \searrow 1,$ the LHS decays to $0$, which means that $f(x) \to 0$ as $x \to 1^+$. This means that either the codomain proposed in the question is incorrect, or that the answer is that there is no solution (which is more boring). Let us suppose that $f: (1,\infty) \to [0,\infty)$ was meant: we can't have the range including negative numbers since then $f(x)^y$ is iffy to define. Note that taking $\log$s over the original functional equation still remains valid, since for any $x > 0, x^{\textrm{anything real}} > 0$.
Suppose that there exist at least two values $y_1, y_2$ such that $f(y_1) > 0$ and $f(y_2) > 0$. Then plugging in $y = y_1$ into the relation, we conclude that for every $x$, $$ f(x) = \left( \frac{f(y_1)^x \log x}{\log y_1}\right)^{1/y_1},$$ and by the same coin, for every $x$, $$ f(x) = \left( \frac{f(y_2)^x \log x}{\log y_2}\right)^{1/y_2}.$$
Equating these, we conclude that for every $x$, $$ \left( \frac{f(y_1)^{y_2}}{f(y_2)^{y_1}}\right)^x \log^{y_1 - y_2} x= \frac{(\log y_1)^{y_2}}{(\log y_2)^{y_1}}.$$ But this is an equation of the form $e^{\alpha x} \log^n(x) = \beta$ for constants $\alpha,n,\beta$ where $\beta > 0,$ which can't hold for every $x$: indeed, wlog, we can assume that $\alpha \ge 0$ (exchange $y_1$ and $y_2$ otherwise), and then no matter the sign of $n$, $x \mapsto e^{\alpha x} \log^n x$ is strictly increasing for large enough $x$, since its derivative is $e^{\alpha x} \log^{n} x ( \alpha + n/x\log x),$ which eventually exceeds $\alpha e^{\alpha x} \log^n(x)/2 > 0$ (since $1/x\log x \to 0$), which means that the equality cannot hold for all $x$ (thanks to Stefan for catching an error in an earlier version of this bit).
We can thus infer that $f$ can have at most one point at which its value is $> 0$. But again if $\exists y_0 > 1 : f(y_0) = c > 0,$ then for every $x \neq y_0,$ $c^x \log x = 0^{y_0} \log y_0 = 0$, which contradicts the elementary fact that $c^x \log x > 0$ over $(1,\infty)$ if $c > 0.$
It follows then that the only candidate is $f(x) =0$ identically. This does satisfy the original functional equation, and we're done.