In Nakahara's "Geometry, Topology and Physics" (and many other sources) the instanton number of an $SU(2)$ instanton $A$ with curvature $F^A$ is defined by
$$\int_{S^4}\text{ch}_2(E)=\frac{1}{2}\int_{S^4}\text{tr}(\frac{iF^A}{2\pi})^2$$ I came across https://esc.fnwi.uva.nl/thesis/centraal/files/f1909314163.pdf which is a thesis that discusses instantons, and on page 75 it performs a calculation that uses $$\frac{1}{4\pi^2}\int_{S^4}|F^A|^2dV_g$$ as the definition for the instanton number. I do not understand how to reconcile these two definitions. By the linearity of the trace operator, Nakahara's definition can be written as $$-\frac{1}{8\pi^2}\int_{S^4}\text{tr}(F^A)^2$$
Could anyone explain to me how these two definitions are equivalent? In particular, it isn't clear to me what $|F^A|^2$ precisely means. I think this is the main source of confusion.
The curvature $F_A$ is locally a 2-form with values in $\mathfrak{su}(2)$. To define its norm we combine the Riemannian metric with the invariant inner product on $\mathfrak{su}(2)$ given by $\langle A, B\rangle = Tr(AB^*) = -Tr(AB)$. Explicitly, if locally $F_A = F_{ij}dx^i\wedge dx^j$ for $F_{ij}\in\mathfrak{su}(2)$, then $$ |F_A|^2 = -Tr(F_{ij}F_{kl})(g^{ik}g^{jl} - g^{il}g^{jk}) $$ This is expressed in coordinate-free terms using the Hodge star: $$ -Tr(F_A \wedge * F_A) = |F_A|^2 dV_g. $$
On the other hand we can write $F_A = F_A^+ + F_A^-$ as the sum of self-dual and anti-self-dual parts. Since a self-dual form wedged with an anti-self-dual 2-form is zero, one finds $$\begin{align} -Tr(F_A\wedge F_A) &= -Tr(F_A^+\wedge F_A^+) - Tr(F_A^- \wedge F_A^-)\\ &=(|F_A^+|^2 - |F_A^-|^2)dV_g \end{align}$$ by replacing $F_A^\pm$ by $\pm * F_A^\pm$ in one factor of each term in the first line.
By orthogonality of $F_A^+$ and $F_A^-$, we have $|F_A|^2 = |F_A^+|^2 + |F_A^-|^2$. Hence $$\begin{align} \frac{1}{8\pi^2}\int|F_A|^2 dV_g &= \frac{1}{8\pi^2}\int (|F_A^+|^2 + |F_A^-|^2)dV_g\\ &\geq \frac{1}{8\pi^2} \int(|F_A^+|^2 - |F_A^-|^2)dV_g\\ &=-\frac{1}{8\pi^2}\int Tr(F_A^2), \end{align}$$ with equality if and only if $F_A^- = 0$, i.e. $F_A$ is self-dual.
The definition from Nakahara is the "right" one, in the sense that it is determined topologically and does not depend on the connection $A$. The above shows that it is equal to the definition in the thesis you cite (modulo a factor of 2) in the case of a connection with self-dual curvature, which seems to be what is under consideration in the thesis. Moreover, the above is the justification of the standard fact that the self-dual instantons are exactly the global minimizers of the Yang-Mills energy $\int |F_A|^2 dV_g$.
Note that the factor of 2 might be accounted for by a different convention on the norm in $\mathfrak{su}(2)$, where it is also natural to set $\langle A, B\rangle = -\frac{1}{2}Tr(AB)$.