Let $h:\mathbb{R}\to\mathbb{R}. $I want to prove that the unique solution of $$h\left(\frac{x}{y}\right)h(y)h\left(\frac{1}{x}\right)=h\left(\frac{y}{x}\right)h(x)h\left(\frac{1}{y}\right)$$ is $h(x)=ax^k$.
My partial solution is: Taking $x=y^2$ is the above equation, we get $$\left(\frac{h(y)}{h\left(\frac{1}{y}\right)}\right)^{2}=\frac{h(y^2)}{h\left(\frac{1}{y^2}\right)}$$ By induction, we get $$\left(\frac{h(y)}{h\left(\frac{1}{y}\right)}\right)^{2^n}=\frac{h(y^{2^n})}{h\left(\frac{1}{y^{2^n}}\right)}$$ this implies that the solution is $ax^k$?
Monomials are not the only continuous solutions, $h(x) = \cos\left(x-\frac{1}{x}\right)$ is a counter-example.
Let's define $\phi(x) = \frac{h(x)}{h(1/x)}$; then, your relation becomes $\phi(x/y) = \frac{\phi(x)}{\phi(1/y)}$, which can be rewritten recursively as $$ \begin{array}{rcl} \phi(x) = \phi(a)\phi(a x) = \phi(a)\phi(b)\phi(abx) = \phi(a)\phi(b)\phi(c)\phi(abcx) = \ldots \end{array} $$ After an infinite number of steps, the left-hand side doesn't depend on $x$ anymore, that is why $\phi(x)$ is a constant $-$ the same argument is used with infinitely nested radicals for instance. Moreover, as the quantities $a,b,c,\ldots$ are arbitrary, the only possibilities are $\phi\equiv0$ and $\phi\equiv1$; the first case doesn't provide any solution for $h$, while the second case leads $h(x) = h(1/x)$, whose solution takes the form $h(x) = S\left(x,\frac{1}{x}\right)$, where $S(x,y)$ is a symmetric function (cf. https://eqworld.ipmnet.ru/en/solutions/fe/fe1120.pdf).
Examples of solutions are thus $\cos\left(x-\frac{1}{x}\right)$, $\ln\left(x+\frac{1}{x}\right)$, $\left(x+\frac{1}{x}\right)^3$, etc. Note that monomials are associated to combinations of polynomial $S(x,y)$. If you want a continuous solution over the whole real line (because $h(1/x)$ is not defined at $x=0$ even for monomials), $h(x) = \exp\left(-\left|x-\frac{1}{x}\right|\right)$ is one of them for example.