The question is as in the title. I have browsed online (Wikipedia, etc) and while they do give me the definition, it gets a bit too much physics-y for me. Therefore I would appreciate it if someone would spoonfeed me the math a little.
I am especially interested in the case of moduli of instantons on a compact complex surface. How does one construct such moduli spaces? How can these objects be used to differentiate between surfaces?
I believe I have a decent enough background in algebraic/complex/differential geometry so a high-tech answer would be appreciated. Thank you!
An instanton is a particular case of connection. Hence the set of all instantons is a subset of the space of connections $\mathcal{C}(E) $ ($E\to X^4$ is our bundle). $\mathcal{C}(E)$ is a topological space, (hence also the set of instantons has a topology), and is acted upon by the gauge group $\mathcal G(E)$ (the group of automorphisms of the bundle). The moduli space $\mathcal M$ of instantons is just the image of in the quotient $\mathcal C(E)/\mathcal G(E)$. Under some assumptions $\mathcal M$ has a natural structure of a smooth manifold, non-compact but with a "nice" compactification. The diffeomorphism class of it is an invariant of the pair $(E,X)$.
Invariants like Donaldson's evaluate some characteristic classees of bundles over $\mathcal M$, hence in particular are homotopy invariant of $\mathcal M$. These are powerful (alas hard to compute) invariants of $(E,X)$ because they depend on the smooth structure of $X$.
For $X$ Kähler we have also the following.
Corollary 6.1.6 of Donaldson-Kronheimer's "The Geometry of 4-manifolds". If $E$ is an $SU(2)$ bundle over a compact Kähler surface $X$, the moduli space of irreducible* instantons is naturally identified, as a set, with the set of equivalence classes of stable holomorphic $SL(2,\mathbb C)$ bundles which are topologically equivalent to $E$.
Maybe you can get some functorial properties out of this perspective, idk.
Also, if you are specifically interested in complex surfaces a good reference is Friedman-Morgan's "Smooth 4-manifolds and complex surfaces".
*meaning connections with trivial stabilizer wrt the gauge group action.