Suppose that $a\in(0,1)$ and that $f:\mathbb R \to \mathbb R$. I am trying to solve for $f$ such that
$$ \big(1+f(x^{-1})\big)\big(f(x)-f(x)^{-1}\big)=\frac{(x-a)(1-ax)}{x} .$$
I really don't know where to start from though. I've tried the method of undetermined coefficients supposing that $f$ is linear and quadratic, but that did not work. I think I need a better guess for a functional form. I am not really looking for the solution, but for some help with where to start from. References are very welcome.
$$$$ $$$$
Suggestion from the comments:
The power series, $$ f(x) = \sum_{i=0}^\infty b_ix^i $$ leads to the following system of equations:
\begin{align} &\;\vdots\\ [x^4] &: \;\;\sum_{i=3}^{\infty}\left(\sum_{j=0}^{i}b_{j}b_{i-j}\right)b_{i-3}=(1+a^2)b_3-a(b_2+b_4)-2(b_0 b_3+b_1 b_2)\\ [x^3] &: \;\;\sum_{i=2}^{\infty}\left(\sum_{j=0}^{i}b_{j}b_{i-j}\right)b_{i-2}=(1+a^2)b_2-a(b_1+b_3)-(b_1^2+2b_0 b_2)\\ [x^2] &: \;\;\sum_{i=1}^{\infty}\left(\sum_{j=0}^{i}b_{j}b_{i-j}\right)b_{i-1}=(1+a^2)b_1-a(b_0+b_2)-2b_0 b_1\\ [x^1] &: \;\;\sum_{i=0}^{\infty}\left(\sum_{j=0}^{i}b_{j}b_{i-j}\right)b_{i}=1+(a^2-b_0)b_0-ab_1\\ [x^0] &: \;\;\sum_{i=0}^{\infty}\left(\sum_{j=0}^{i}b_{j}b_{i-j}\right)b_{i+1}=b_1-ab_0\\ [x^{-1}] &: \;\;\sum_{i=0}^{\infty}\left(\sum_{j=0}^{i}b_{j}b_{i-j}\right)b_{i+2}=b_2\\ [x^{-2}] &: \;\;\sum_{i=0}^{\infty}\left(\sum_{j=0}^{i}b_{j}b_{i-j}\right)b_{i+3}=b_3\\ &\; \vdots \end{align}
Hint (while assuming $x \ne 0, f(x) \ne 0$ etc) ... First note that the RHS is symmetric with respect to $x \mapsto \frac{1}{x}$ which becomes obvious if rewritten as $\frac{(x-a)(1-ax)}{x} = (x-a)(\frac{1}{x}-a)$. This suggests taking the $x \mapsto \frac{1}{x}$ counterpart of the identity and equating the two to get:
$$ \begin{align} & \left(1+f\left(\frac{1}{x}\right)\right) \left(f(x)-\frac{1}{f(x)}\right) = \Big(1+f(x)\Big) \left(f\left(\frac{1}{x}\right)-\frac{1}{f\left(\frac{1}{x}\right)}\right) \\ \iff \quad & \frac{f(x) \big(f(x)+1\big)}{\big(f(x)\big)^2 -1} = \frac{f(\frac{1}{x}) \big(f(\frac{1}{x})+1\big)}{\big(f(\frac{1}{x})\big)^2 -1} \\ \iff \quad & \frac{f(x)}{f(x) -1} = \frac{f(\frac{1}{x})}{f(\frac{1}{x})-1} \\ \iff \quad & f(x) = f(\frac{1}{x}) \end{align} $$
Then, with $y = f(x) = f(\frac{1}{x})$, the original identity becomes:
$$\Big(1+y\Big)\Big(y - \frac{1}{y}\Big) = \Big(x-a\Big)\Big(\frac{1}{x} - a\Big)$$
which gives a cubic in $y$ that can be solved to obtain a closed form for $y=f(x)$ though it's not obvious it would be a "nice" one.