I come up with this interesting question in Serge Lang's Algebra:
Let $k$ be a field of $q$ elements and $K=k(X)$ be the rational field of one variable. Let $G$ be the group of automorphisms obtained by the mappings $$ X \mapsto \frac{aX+b}{cX+d} $$ with $a,b,c,d \in k$ and $ad-bc \ne 0$.
(a) Show that $G$ has $q^3-q$ elements.
(b) Show that the fixed field of $K$ is $k(Y)$ with $Y=\frac{(X^{q^2}-X)^{q+1}}{(X^q-X)^{q^2+1}}$.
(a) is easy because it's basically counting the cardinality of $PGL_2(k)$.
My question is (b) however. Put $Y=Y(X)=\frac{f(X)}{g(X)} \in k(X)$, we need to show that $Y\left(\frac{aX+b}{cX+d}\right)=Y(X)$. However I can't see how to proceed other than unfeasibly computing all coefficients with even no knowledge of the degree of $f$ and $g$. Is there any properties of $K$ related to this exercise that I have missed?
Edit 1: I've noticed that $[k(X):k(Y)]=q^3+q^2>|G|=q^3-q$. It is also clear that $Y$ is fixed by $G$ (just verify it on all generators of $G$). Perhaps then the problem can be narrowed down to a simple Galois theory problem?
You miscalculated the extension degree $[k(X):k[Y)]$. Regardless of whether the extension is Galois or not, we always have $|G|\le|\operatorname{Aut}_{k(Y)}(k(X))|\le [k(X):k(Y)]$ due to the independence of characters. (See Theorem 13, Section II.F. of Artin's Galois Theory. In fact, the case of $q=2$ was treated in Section II.G.)
Note that the polynomial $T^q-T$ has no repeated root (since its derivative is $-1$ which itself has no root), and any root also satisfies $T^{q^2}-T=0$ (if $T^q=T$, then $T^{q^2}=(T^q)^q=T$). Therefore $T^q-T|T^{q^2}-T$ as polynomials. In particular, we know that the denominator and the numerator of $Y$ as presented are not coprime.
Indeed, after cancelling $(X^q-X)^{q+1}$, the degree of the denominator is $q^2(q+1)-q(q+1)=(q^2-q)(q+1)=q^3-q$, and the one of the numerator is $q(q^2+1)-q(q+1)=q(q^2-q)=q^3-q^2<q^3-q$. From this, we know that the minimial polynomial of $X$ in $k(Y)$ is of degree at most $q^3-q$. Hence the earlier inequalities become equalities $|G|=|\operatorname{Aut}_{k(Y)}(k(X))|=[k(X):k(Y)]$, we must have $G=\operatorname{Aut}_{k(Y)}(k(X))$ and the extension is Galois.