Let $X$ be a compact Hausdorff space, and $H(X)$ be the set of homeomorphisms from $X$ to $X$, with the compact-open topology.
Prove that the mapping $h:H(X)\times H(X)\rightarrow H(X)$, $h(f,g)=f\circ g$ is continuous.
Note, if $C(X,X)$ is the set of all continuous mappings from $X$ to $X$, the compact-open topology on $C(X,X)$ is generated by subsets of the form $B(K,U)=\{f:f(K)\subset U\}$ where $K$ is compact in $X$ and $U$ is open in $X$.
I honestly have no clue how to work with the compact open topology and would appreciate any hints.
Let's take $U$ open in $H(X)$. I want to show that $h^{-1}(U)$ is open in $H(X)\times H(X)$.
I believe I am overthinking this, and apologize for the lack of work, I am just really confused how to show this. Any help would be much appreciated.
We will use the characterization of continuity that says that a map $f:X\rightarrow Y$ is continuous if for all $x\in X$ and open $U\subseteq Y$ such that $f(x)\in U$ we have that there is an open $V\subseteq X$ such that $x\in V$ and $f(V)\subseteq U$.
Let $B(K,U)\subseteq H(X)$ be given where $K\subseteq X$ is closed (compact) and $U\subseteq X$ is open. Then, if $g\circ f\in B(K,U)$ we have that $gf(K)\subseteq U$. Because $g$ is continuous we have that $g^{-1}(U)$ is open in $X$ can contains $f(K)$. Because $X$ is normal (recall that compact Hausdorff spaces are normal) there is an open $V\subseteq X$ such that $f(K)\subseteq V\subseteq\overline{V}\subseteq g^{-1}(U)$. Because $X$ is compact we have that $\overline{V}$ is compact. Moreover it is clear that $g(\overline{V})\subseteq U$. We then claim the following:
$$(B(\overline{V},U)\circ B(K,V))\subseteq B(K,U)$$
To see this we simply let $(k,l)\in B(\overline{V},U)\times B(K,V)$. Then, by definition $l(K)\subseteq V$ and $k(\overline{V})\subseteq U$. Then we can easily see that $(k\circ l)(K)\subseteq U$. It is also clear that $g\circ f\in B(\overline{V},U)\times B(K,V)$. Therefore composition is continuous.
Note: This result generalizes quite easily to the following situation as seen in an exercise of Munkres. Let $Y$ be locally compact Hausdorff, and $X$ and $Z$ general spaces. Also let $\mathcal{C}(X,Y),\,\mathcal{C}(Y,Z),$ and $\mathcal{C}(X,Z)$ denote the spaces of continuous functions from the respective spaces with the compact open topology. Then the composition map
$$\mathcal{C}(X,Y)\times\mathcal{C}(Y,Z)\rightarrow\mathcal{C}(X,Z)$$
is continuous.