let G be the subset of $\operatorname{aut}_{K} K(x)$ consisting of the three automorphisms
$$ x \mapsto x $$ $$ x \mapsto 1/(1-x)$$ $$ x \mapsto (x-1)/x$$
then G is a subgroup of $\operatorname{aut}_K K(x)$. determine the fixed field of $G$
solution:(wrong) let $ f/g \in K(x)$ with $f$ and $g$ relatively prime in K[x], and suppose that $f/g$ is in the fixed field then $ f/g = 1/(1-(f/g))$ which gives $ f^2 -fg=g^2$
$g^2 = f(g-f)$ so we have that $ f \mid g^2$ so we must have that $f$ is a constant since $f$ and $g$ are relatively prime. and by symmetry we have that $g$ must be a constant which is a contradiction so we must have that $f/g$ is not in the fixed field of G, but this is true for every $f/g \in $ K(x) so the fixed field of G must be empty.
is that right im not sure if that is a contradiction or not?
new solution:
let $\dfrac{ax+b}{cx+d} \in Aut_{K}K(x)$ $$ \sigma_{1} :x \mapsto x $$ $$ \sigma_{2} :x \mapsto 1/(1-x)$$ $$ \sigma_{3} :x \mapsto (x-1)/x$$
$\dfrac{ax+b}{cx+d} = x$ gives $ax+b=cx^2+dx$, which gives $c=0$, $b=0$, and $a=d$ or $(a,b,c,d)= (a,0,0,a)$
$\dfrac{ax+b}{cx+d} =1/(1-x)$ gives $ax+b-ax^2-bx =cx+d$ , gives $a=0$, $b=d$, and $c=-b$ or $(a,b,c,d)=(0,b,-b,b)$
$\dfrac{ax+b}{cx+d} = (x-1)/x$ gives $ax^2+bx=cx^2+dx-cx-d$ gives $a=c$, $d=0$, $b=-c$ or $(a,b,c,d)= (a,-a,a,0)$
This is not a complete answer, but it might get you started.
Let's call the three given automorphisms $1,\sigma,\tau$. Convince yourself that $\sigma^2=\tau$ and $\sigma\tau=1$. Now let $f$ be any element of $K(x)$. Can you see that $f+\sigma f+\tau f$ is in the fixed field of $G$?
For example, if we just take $f=x$, then $$f+\sigma f+\tau f=x+{1\over1-x}+{x-1\over x}={x^3-3x+1\over x(x-1)}$$ so $(x^3-3x+1)/(x(x-1))$ is in the fixed field of $G$.