In the question, I'm asked to show that \begin{align*} F\begin{pmatrix}x\\y\end{pmatrix}=x^2+y^2 \end{align*} is an integral for the linear map \begin{align*} L(\text{x})= \begin{pmatrix} 0&1\\-1&0 \end{pmatrix} \text{x}. \end{align*}
I know that to solve the exercise I need to show that F is constant along the orbits of L. That is, I need to show that $F ∘ L(x) = F(x)$.
My question is how to you compose the two in this case? In $F\begin{pmatrix}x\\y\end{pmatrix}$ Do you set both variables equal $L(x)$ or only one....
The following is based on the assumption that one considers a dynamical system in continuous time. It appears that the object of interest is actually a dynamical system in discrete time (aka a recursive sequence...) hence the answer below does not apply. Leaving it for the picture...
Note that the question actually asks why $x^2+y^2=(-y)^2+x^2$ (?!), which makes for a rather poor question, I am sorry to say. Note also that every sequence $(\mathbf x_n)$ defined by $\mathbf x_{n+1}=L(\mathbf x_n)$ has period $4$, as an elementary computation shows.
No. That $F$ is an invariant of the dynamical system associated to $L$ means that, if $$\dot{\mathbf x}(t)=L(\mathbf x(t)),$$ then the quantity $F(\mathbf x(t))$ does not depend on $t$. This is usually done showing that $u'(t)=0$ for every $t$, where $$u(t)=F(\mathbf x(t)),$$ which is implied by the condition that, for every $\mathbf x$, $$\nabla F(\mathbf x)\cdot L(\mathbf x)=0.$$ In your case, $$\nabla F(\mathbf x)=2\mathbf x,$$ hence one is left with $$\nabla F(\mathbf x)\cdot L(\mathbf x)=2\mathbf x\cdot L(\mathbf x)=2\begin{pmatrix}x\\y\end{pmatrix}\cdot\begin{pmatrix}y\\-x\end{pmatrix}=0.$$ Here is a rough sketch of some solutions $(\mathbf x(t))$:
$\hspace2in$