Let $n\in 2\mathbb Z$ be an even number. Let $G=\operatorname{GL}_n(\mathbb{C})$ and $V_\lambda$ the irreducible complex $G$-module corresponding to the partition $\lambda=(\lambda_1\ge\cdots\ge\lambda_n)$. Let $$H:=\operatorname O_n(\mathbb C) :=\{ g\in G \mid g^\intercal g = \operatorname{id} \}$$ be the orthogonal group. I would like to know under what conditions $V_\lambda$ contains nonzero $H$-invariant vectors. Differently put, when do we have $V_\lambda^H \ne \{0\}$? This should be a condition on the partition $\lambda$ only.
I would also be thankful for any information on how $G$-representations restrict to $H$ in general. I feel that this is probably well-known, but I have no reference for it. Thanks a lot!
The branching rule of $\mathrm O_n \subset \mathrm{GL}_n$ gives $$V_\lambda = \bigoplus_{\mu} \Big(\sum_{\delta} c_{2\delta,\mu}^\lambda \Big) W_\mu $$ where $W_\mu$ are the irreducible $\mathrm O_n$ representations and $c$ the Littlewood-Richardson coefficients. (Note that $2\delta$ are "even partitions", i.e. all summands are even.)
Your question is now, for which $\lambda$ is $$\Big(\sum_{\delta} c_{2\delta,(0)}^\lambda \Big)\neq 0?$$ From some very basic facts about the Littlewood-Richardson coefficients, we know $2\delta =\lambda$ for any $c_{2\delta,(0)}^\lambda \neq 0$, thus $\lambda$ is an even partition.
References for the branching rules are plenty. For instance Fulton-Harris p.427 (25.37), or a good reference on all kinds of branching rule is R. Howe, E-C Tan, and J. Willenbring: Stable branching rules for classical symmetric pairs, Transactions of the American Mathematical Society Volume 357, Number 4, Pages 1601–1626 (Here it's Theorem 1.1)