Let $G$ be a compact topological group with Haar measure $dg$. We can decompose a finite-dimensional complex representation as $V\cong \mathbb C^k\oplus V_1\oplus\dots\oplus V_n$, where $G$ acts trivially on $\mathbb C^k$ and each $V_i$ is a nontrivial, irreducible represetation. Then the orthonormality of irreducible characters yields $$\int_G \chi_Vdg=\int_G (k+\sum\chi_{V_i})dg=k+\sum\int_G \chi_{V_i}\overline{\chi_{\mathbb C}}dg=k=\dim(V^G),$$ the dimension of the fixed-point subspace. But if we start with the fact that $\int_G \chi_Vdg=\dim(V^G)$, we can see that $$\int_G \chi_V\overline{\chi_W}dg=\int_G\chi_{V\otimes W^*}dg=\int_G\chi_{\text{Hom}(V,W)}dg=\dim\text{Hom}_G(W,V).$$ By Schur's lemma, this dimension is either 1 or 0, depending on whether or not the representations $V$ and $W$ are isomorphic. This argument therefore reproduces the orthonormality of irreducible characters.
But to follow the second line of reasoning, we need to start with some other proof that $\int_G \chi_Vdg=\dim(V^G)$. This makes me wonder if there is another proof of this formula, which does not make any recourse to orthogonality of irreducible characters or matrix entries (as this would seem—as best I can tell—more like a re-worded version of the same proof)?
Following Pedro's comment, here is an explanation exactly of the sort that I was hoping for:
Let $\rho:G\rightarrow \text{GL}(V)$ be a representation. Then we can get an operator $T\in \text{GL}(V)$ by averaging over $G$: $$T=\int_G\rho(g)\,dg$$ This operator clearly fixes every vector in $V^G$ and we get $T(V)\subseteq V^G$ from the left-invariance of the Haar measure. Thus $T$ is a projection onto $V^G$ and so $\text{tr}\,(T)=\dim(V^G).$ This implies that$$\int_G\chi_V(g)\,dg=\int_G\text{tr}\big(\rho(g)\big)\,dg=\text{tr}\left(\int\rho(g)\,dg\right)=\text{tr}\,(T)=\dim(V^G).$$