Magma (link) has a lot of functionality for computing quotients of curves by group actions. I am interested to know how one does this in general and I am finding it oddly difficult to find literature or textbooks that deal with how to do this in practice.
The case I am mainly interested in is where the curve is hyperelliptic over a ground field with positive characteristic and the group action is (finite) a subgroup of the group of automorphisms of the curve (also the case where the characteristic divides the order of the group).
(All I have found so far was in (bizarrely) Milne's book on the étale cohomology that seemed to suggest that one computes the ring of invariants of the function field of the curve and this is the function field of the quotient, but I would like to see some literature all the same.)
The fact that you want to look at the invariants is a bit tautologic: given a group $G$ acting on a curve $X$, any function on $X/G$ lifts to a function on $X$ which is invariant by the action of $G$. For more details about computing the invariant algebra, you might want to look at Bernd Sturmfels' book on the topic. Spoiler: this will involve Gröbner bases...
The reason why data on this subject lives in an étale cohomology book is because in general, quotients are hard in algebraic geometry (just as in any other geometry actually). Namely, they have a tendency to produce ugly singularities (for an easy example, consider the case of the quotient of your hyperelliptic curve by its hyperelliptic involution - this produces singularities at each fixed point).