Let $G$ be a compact Lie group and let $V$ be a finite dimensional real vector space with a faithful action of $G$ given by a Lie group embedding $G\hookrightarrow GL_\mathbb{R}(V)$. Then the orbits of $G$ in $V$ are closed, because they are compact, because they are the images of the compact space $G$ under the continuous map $g\mapsto gv$ ($v\in V$ fixed).
Meanwhile, $G$ can be viewed as an algebraic group defined over $\mathbb{R}$ (in fact, a Zariski-closed subset of $GL_\mathbb{R}(V)$). (I learned this from Onischchik and Vinberg's book Lie Groups and Algebraic Groups, Ch. 3, Sec. 4, Thm. 5.) Then the action map $G\times V\rightarrow V$ is a polynomial map defined over $\mathbb{R}$ because multiplication of a matrix by a vector is polynomial in the coordinates.
The complexification $G_\mathbb{C}$ of $G$ is reductive, and every reductive algebraic group over $\mathbb{C}$ arises this way. (Again, Onishchik and Vinberg, this time Ch. 5, Sec. 5, Thm. 12.)
In general, $G_\mathbb{C}$ need not act on a $\mathbb{C}$-vector space with closed orbits. For example if $G=S^1$, the circle group, then $G_\mathbb{C}$ is $GL_1(\mathbb{C})$, whose action by scaling on $\mathbb{C}^n$ famously has the origin contained in the closure of every orbit.
However, what specifically about the action of $G_\mathbb{C}$ on $V_\mathbb{C}:=\mathbb{C}\otimes_\mathbb{R} V$, defined just by base-changing the original action map $G\times V\rightarrow V$ to $\mathbb{C}$?
Are the orbits of $G_\mathbb{C}$ on $V_\mathbb{C}$ closed?
In summary, if the action of a reductive algebraic group over $\mathbb{C}$ on a $\mathbb{C}$-vector space arises by complexifying the action of its compact real form on a real vector space, does it have closed orbits?
(I think the answer should be yes, based on working out the case of $S^1\cong SO(2)$'s canonical action on $\mathbb{R}^2$ by hand, and some hand-wavy logic involving the fact that the action map is defined over $\mathbb{R}$, but it's very far from a real argument. Thank you in advance for your help.)
Addendum: As Moishe Kohan notes in comments, my conclusion for $SO(2)$ was actually wrong. Even for this action, the orbits are not all closed. (I've written this up as an answer in order to have something to accept.) This is probably not of interest except to me, but, here I will record how I came to this false conclusion:
- I viewed $G$'s variety structure as coming from the ring $\mathbb{R}[x,y]/(x^2+y^2-1)$.
- Then $G_\mathbb{C}$'s variety structure comes from this ring tensored up to $\mathbb{C}$, which is isomorphic to $\mathbb{C}[t,t^{-1}]$ via $t=x+iy$.
- Then, I parametrized the orbits of the action in terms of $t$. They had coordinates with shapes like $(t+t^{-1})u+i(t-t^{-1})v$.
- I reasoned that as $t\to 0,\infty$, one of $t$ or $t^{-1}$ will always blow up, so these coordinates will always blow up, except if $u=v=0$. (False! See next bullet.) Thus the nonzero orbits all go off to infinity as $t\to 0,\infty$, so how will we get any limit points from $t\to 0,\infty$? So the orbits are closed. (Again, false, see below.)
- While this works for most orbits, I didn't consider that $u$ and $v$ can be chosen to make either the $t$'s or the $t^{-1}$'s cancel out. So special choices of $u$ and $v$ will have coordinates with shapes like $t$ or $t^{-1}$, where you do get 0 as a limit point (as $t\to 0,\infty$ respectively). [In fact, the special choices of $u$ and $v$ are the eigenvectors for the action.]
Moishe Kohan notes in comments that the answer is no, even on the basis of the example mentioned in the OP.
The complexification of the canonical action of $S^1$ on $\mathbb{R}^2$ is isomorphic to the action of $\mathbb{C}^\times$ on $\mathbb{C}^2$ via the diagonal matrices
$$\begin{pmatrix}\alpha & \\ &\alpha^{-1}\end{pmatrix}, \;\alpha\in\mathbb{C}^\times.$$
The rotation matrices are simultaneously diagonalizable over $\mathbb{C}$, with eigenvectors $(1,i)$ and $(1,-i)$. The rotation through $\theta\in [0,2\pi)$ has eigenvalues $e^{i\theta}$ and $e^{-i\theta}$ respectively. So letting $\alpha = e^{i\theta}$, we get the action just described. I am omitting details.
This action does not have closed orbits. Most of the orbits are closed, but two of the orbits are the two one-dimensional linear spaces spanned by the eigenvectors of the action, minus the origin. So even in this instance, the action of $G_\mathbb{C}$ does not have closed orbits.