I have a few question regarding the Haar measure on the orthogonal, or unitary, or indeed Lie, groups. Let me consider $SO(N)$.
If I were to define a probability measure on $SO(N)$, I would proceed like this. Every such matrix is the exponential of a real antisymmetric matrix, $O=e^M$. The antisymmetry constraint is much simpler than the orthogonality constraint, and I could simply put a Gaussian measure on the off-diagonal elements of $M$, thereby inducing a probability measure on $SO(N)$.
First question: Is the Haar measure on $SO(N)$ related at all to the above procedure?
Also, I have seen in the literature the expression $d\mu(O)=O^TdO$ to indicate the Haar measure, with the argument that this is invariant under adjoint action since $O\mapsto AO$ leaves it invariant as $(AO)^Td(AO)=O^TA^TAdO=O^TdO$.
Fine. But, second question: What is meant by $dO$ in the above notation?
If $O=e^M$, then I suppose $dO=OdM$ and then $O^TdO=dM$. Does this make any sense?
Disclaimer: This is more of an extended comment.
In the limiting case that your Gaussian measure has nearly zero variance, the measure you are constructing is concentrated near the identity matrix, and cannot be the Haar measure. In the other limit, as the variance tends to infinity, you do recover the Haar measure.
So what's going on? This process you described is one way to construct the heat kernel on $SO(N)$. If you like, you can imagine you are doing a Brownian motion on the antisymmetric matrices, and turning that into a Brownian motion on $SO(N)$ via the exponential map. As $SO(N)$ is compact, there is a finite mixing time, so you will get rapid convergence to the Haar measure as the variance increases.
I hope that says something about your first question.
I have a guess is to what is meant by $O^TdO$, and it is very similar to your $OdM$, except that one doesn't really need to explicitly pick a logarithm $M$.
Let $O\in SO(N)$ and let $A\subset SO(N)$ be a set with $A\subset B_\epsilon(O)$. Then $O^TA$ is a set of matrices that is very close to being antisymmetric perturbations of the identity. To approximate $\mu(A)$, one can project onto this set and use the Lebesgue measure on the antisymmetric matrices. The point is, I think the $dO$ is supposed to refer to the Lebesgue measure, and perhaps this is the sort of thing they mean.
After all, it doesn't make sense to evaluate a measure at a point, but this is often shorthand for the density of the measure on small sets near that point, and I just tried to unpack it.