Consider the following integration $$\int_\phi R dR $$ where $\phi$ is a set of all orthonormal matrices. In case the above integration doesn't make sense, what I want to evaluate is the following:
$$\frac{1}{|\phi|} \sum_{R\in \phi } R $$
My guess is that the answer is a matrix of 0s, but I am not sure how to prove it.
The problem does make sense, but to make sense of it you need to have a measure on the space of orthogonal matrices. The standard choice if you want to integrate over a compact Lie group is to take $\mu$ to be the left (or right) invariant Haar measure such that the total group has measure 1. So $\mu(S)=\mu(gS)$ for every measureable set $S$ and $g\in G$. If you have a Haar measure and a measurable function $f$, then translation invariance of the measure implies that
$$\int_G f(x)d\mu(x)=\int_G f(g^{-1}x) d\mu(x)$$
for any $g\in G$. Now, if $\rho$ is a representation of $G$, so that we are integrating matrices (which we can view as coordinatewise integration), we can use this translation invariance.
$$\int_G \rho(x)d\mu(x)=\int_G \rho(g^{-1}x)d\mu(x)=\int_G \rho(g)^{-1}\rho(x)d\mu(x)=\rho(g)^{-1}\int_G \rho(x)d\mu(x).$$
So the integral, whatever it is, will also be preserved by multiplication with anything in the image of $\rho$.
Taking $G=O(n)$, $\rho$ the standard representation on $\mathbb R^n$, and $g=-I$ is enough to see the integral must be $0$.