From Milne's Fields and Galois Theory, I try to understand the following application of the main theorem of Kummer Theory (page 73):
As you can see from my notes, I do not really understand where the $\infty$ comes from. Up to this part, I understood everything (in particular, why the finite abelian extensions of $\mathbb{Q}$ of exponent $2$ correspond to the subgroups of $\mathbb{Q}^\times$ which contain $\mathbb{Q}^{\times 2}$ as a subgroup of finite index, and why every nonzero rational number has a unique representative of the form $\pm p_1 \cdots p_r$). I also think this is relevant for the "$-1$" following later in the text, so it seems to be important to understand.
Could you please elaborate on that part?

You have representitives for $\mathbb{Q}^\times /(\mathbb{Q}^{\times})^2$ of the form $\pm p_1 ... p_n$. Then \begin{align*} \mathbb{Z}/2\mathbb{Z} \oplus \bigoplus_{p \text{ prime}} \mathbb{Z}/2\mathbb{Z} \cong \mathbb{Q}^\times /(\mathbb{Q}^{\times})^2 \\ \end{align*} where an $a$ in the first $\mathbb{Z}/2\mathbb{Z}$ is sent to $(-1)^a$ and an $a$ in the $p^{th}$ coordinate is sent to $p^a$.
We see you need an "extra" index to account for the $-1$, since it is not prime - you could simply call this index $-1$. It fits with convention to call this $\infty$ since in algebraic number theory we often talk about primes corresponding to valuations, these give rise to (equivalence classes of) absolute values. The only absolute value not coming in this way is the euclidean one. Going back to the primes we get an extra one - which we call $\infty$.
Edit: Thinking about this more, the naming of this as $\infty$ makes even more sense for the following reason. The primes that ramify in $\mathbb{Q}(\sqrt{\pm p_1...p_n})$ are precisely $p_1, ..., p_n$ and possibly $2$. The place at $\infty$ ramifies if and only if the sign is negative.