Question
Find all functions $f:\ (0, \infty)\ \longrightarrow\ (0, \infty)$ such that
a) $\forall x \in (0,1):\ f(x) \in (1, \infty).$
b) $ \forall x,y \in (0, \infty) :\ f(xf(y))=yf(x)$
My try
I first guessed that function is $1/x$...
Now taking $x=1$ we get $f(f(y))=yf(1)$
Now using this and the fact that $f(1)$ cannot equal $0$, I proved that $f$ is injective. Hence taking $y=1$ above I get $f(1)=1$.
This also implies that $f(f(y))=y$. Using this I proved multiplicativity of $f$, and then I get no where. So I saw the hint and there they have written that:
Using multiplicativity of $f$ prove that $f(\tfrac1x)=\tfrac{1}{f(x)}$ and so $f$ is strictly decreasing in $(0, \infty)$.
I do not understand why these both things are true. How from multiplicativity of $f$ we can show that $f(\tfrac1x)=\tfrac{1}{f(x)}$?
I've written up a full solution for completeness sake; I understand you have already shown the first half yourself.
Plugging in $x=1$ shows that for all $y>0$ we have $$f(f(y))=yf(1).\tag{1}$$ Let $c:=f(1)>0$ so that the function $$g:\ \Bbb{R}_{>0}\ \longrightarrow\ \Bbb{R}_{>0}:\ y\ \longmapsto\ cy,$$ is a bijection. Equation $(1)$ shows that $g=f\circ f$, and so $f$ is also a bijection.
Set $c:=f^{-1}(1)$ so that for all $x>0$ we have $$f(x)=f(xf(c))=cf(x),$$ which shows that $c=1$, so $f(1)=1$. Then it follows that for all $x>0$ also $$f(f(x))=f(1\cdot f(x))=x\cdot f(1)=x.$$ Now for $x,y>0$ we have $y=f(f(y))$ and hence the functional equation shows that $$f(xy)=f(xf(f(y)))=f(y)f(x),$$ which means that $f$ is multiplicative. In particular for $y=x^{-1}$ we get that $$f(x)f(x^{-1})=f(x\cdot x^{-1})=f(1)=1,$$ so indeed $f(x^{-1})=f(x)^{-1}$. Then for $x>y>0$ we have $0<\tfrac{y}{x}<1$ and hence $f(\tfrac yx)>1$ by assumption, where $$f(\tfrac yx)=f(y)f(x^{-1})=\frac{f(y)}{f(x)}.$$ This shows that $x>y$ implies $f(y)>f(x)$, or in other words, that $f$ is strictly decreasing.
Because $f$ is strictly decreasing it has at most one fixed point, and we already saw that $f(1)=1$, so this is the unique fixed point. Now plugging $x=y$ into the functional equation yields $$f(xf(x))=xf(x),$$ for all $x>0$, so $xf(x)$ is a fixed point for all $x>0$. This proves that $xf(x)=1$, or equivalently that $f(x)=\tfrac1x$ for all $x>0$.