I am wondering whether the group of all autohomeomorphisms of a compact metric space can be given a reasonable topological group structure? (Preferably, can it be turned into a locally compact group?)
I think that moral reason should be no, but I might be wrong. Here's how I see it: Boolean algebras, which are discrete in nature correspond to compact, Hausdorff zero-dimensional spaces. The group of automorphisms of a Boolean algebra is then the same as the group of automorphisms of the corresponding Stone space...
Yep, we normally give the automorphism group of a topological space the compact open topology. It's an important object, and is part of the definition of the ubiquitous mapping class group.
The automorphism group of a general topological space tends to be extremely large, which is why we normally look at the mapping class group which is a homotopic analogue and tends to be more manageable.
I'll also add that we don't need to restrict ourselves to the topological category because, as pointed out above, the automorphism group tends to be unmanageable. If we restrict ourselves to metric spaces for instance, and consider the group of bijective autoisometries, the groups tend to be much more well-behaved. For instance the circle $S^1$ has automorphism group (in $\mathbf{Met}$) isomorphic to $S^1\ltimes\mathbb{Z}_2$ or sometimes called $Dih(S^1)$, the circular dihedral group, with the topology of $S^1\sqcup S^1$ inherited in the obvious way.