This is an old exam question which most of it I understand now due to the comments below. I still have two concerns.
Define a $p$-form on $GL(n)$ as follows. $$\Theta_p=tr(X^{-1}dX \wedge X^{-1}dX \wedge \cdots \wedge X^{-1}dX)$$ (i) Restrict $\Theta_3$ to $O(3)$. Writing the matrix $X$ with orthonormal column vectors $\mathbf{x_1, x_2,x_3}$, show that $X^{-1}dX$ is a skew symmetric matrix whose $i,j$th entry is $\mathbf{x}_i\cdot d\mathbf{x}_j$.
(ii) Show that at $X=I$, the forms $\mathbf{x_1} \cdot \mathbf{dx_2}, \mathbf{x_2 \cdot dx_3},$ and $\mathbf{x_3, \cdot dx_1}$ are linearly independent and $\Theta_3$ is nonzero.
(iii) Deduce that since $(AX)^{-1}d(AX)=X^{-1}dX$, $\Theta_3$ is non vanishing at all points.
(a) why can we just restrict $\Theta_3$ to $O(3)$? How do we know this form is still smooth? i.e. the induced map to $O(3) \rightarrow \wedge^*T^*O(3)$ is still smooth?
(b) Why does the equality in (iii) show $X^{-1}dX$ is left invariant?
I am also unsure what this means, since we are working with $1$-forms rather than vector fields. I suppose we are to show: $$(L_{A^{-1}}^*) X^{-1}dX_I = X^{-1}dX_A $$
Yes, the wedge-product on $\Omega^\bullet(GL(n;\mathbb{R}))$ (or $\mathbb{C}$) and the matrix product on $\mathbb{R}^{n\times n}$ together defines a product on the tensor product of rings $\Omega^\bullet(GL(n;\mathbb{R}))\otimes_{\mathbb{R}}\mathbb{R}^{n\times n}$ which is denoted also by $\wedge$, and your formula is correct. Here $X\colon GL(n;\mathbb{R})\to\mathbb{R}^{n\times n}$ is just the inclusion.
(1) $\Theta_p$ is a $p$-form on $GL(n)$ so it restricts (actually pulls back) to $O(n)\subset GL(n)$ under the usual inclusion. Since $X^{-1}=X^T$ for $X\in O(3)$, the form $X^{-1}\,\mathrm{d}X$ is $$ X^{-1}\,\mathrm{d}X=(\mathbf{x}_1\,\mathbf{x}_2\,\mathbf{x}_3)^T(\mathrm{d}\mathbf{x}_1\,\mathrm{d}\mathbf{x}_2\,\mathrm{d}\mathbf{x}_3)=(\mathbf{x}_i^T\mathrm{d}\mathbf{x}_j)=(\mathbf{x}_i\cdot\mathrm{d}\mathbf{x}_j). $$ To show this is skew-symmetric, remember $(X^T)\,\mathrm{d}X+\mathrm{d}(X^T)X=0$ on $O(3)$. Taking the $(i,j)$th component yields: $$ 0=\sum_k(X^T)_{ik}(\mathrm{d}X)_{kj}+(\mathrm{d}X^T)_{ik}X_{kj} =\sum_kX_{ki}\,\mathrm{d}X_{kj}+X_{kj}\,\mathrm{d}X_{ki} =\mathbf{x}_i\cdot\mathrm{d}\mathbf{x}_j+\mathbf{x}_j\cdot\mathrm{d}\mathbf{x}_i. $$
(2) At identity, $\mathbf{x}_i\cdot\mathrm{d}\mathbf{x}_j=\mathrm{d}X_{ij}$, so the three forms are linearly independent. We have $\Theta_3$ is $$ \sum_{i,j,k}(\mathbf{x}_i\cdot\mathrm{d}\mathbf{x}_j)\wedge(\mathbf{x}_j\cdot\mathrm{d}\mathbf{x}_k)\wedge(\mathbf{x}_k\cdot\mathrm{d}\mathbf{x}_i) $$ the summands are nonzero only if $i,j,k$ pairwise distinct, so this is sum over permutations of $1,2,3$. Show that $(\mathbf{x}_i\cdot\mathrm{d}\mathbf{x}_j)\wedge(\mathbf{x}_j\cdot\mathrm{d}\mathbf{x}_k)\wedge(\mathbf{x}_k\cdot\mathrm{d}\mathbf{x}_i)$ is invariant under all permutations (which you should be able to do). So we conclude, at identity, $$ \Theta_3=6(\mathbf{x}_1\cdot\mathrm{d}\mathbf{x}_2)\wedge(\mathbf{x}_2\cdot\mathrm{d}\mathbf{x}_3)\wedge(\mathbf{x}_3\cdot\mathrm{d}\mathbf{x}_1) \neq 0. $$
(3) Since $(AX)^{-1}\,\mathrm{d}(AX)=X^{-1}A^{-1}A\,\mathrm{d}X=X^{-1}\,\mathrm{d}X$, the matrix of $1$-forms $X^{-1}\,\mathrm{d}X$ is left-invariant (on $GL(n)$). Hence $\Theta_3$ is invariant on $GL(3)$ and hence on $O(3)$. Together with (2), this shows $\Theta_3$ must be nonvanishing everywhere on $O(3)$.