Problem:
$\{A,B,E,R,S\}\in R^{n \times n}$ are square matrices, $\{\mathbf{x},\mathbf{y}\}\in R^{n}$ are vectors. Particularly, $\{ A,B \}$ are symmetric matrices, and $E$ is an identical matrix. We consider the equation using these tensors:
$$\biggl[A - \biggl( \cfrac{\mathbf{x}^{\text{T}} A^2 \ \mathbf{x}}{\mathbf{x}^{\text{T}} \mathbf{x}}\biggr)E \ \biggr]\mathbf{x} =\biggl[B-\biggl( \cfrac{\mathbf{y}^{\text{T}} B^2 \ \mathbf{y}}{\mathbf{y}^{\text{T}} \mathbf{y}}\biggr) E \ \biggr]\mathbf{y}$$
Additionally, $A$ and $B$ have a relation which can be expressed by the following equation, which is a kind of affine transformation:
$$B=AR+S$$
Now we know all variables except $\mathbf{x}$. If possible, I want to solve $\mathbf{x}$ analytically without numeric calculations (i.e. Newton's method).
Knowledge:
$\cfrac{\mathbf{x}^{\text{T}} A^2 \mathbf{x}}{\mathbf{x}^{\text{T}} \mathbf{x}}$ and $\cfrac{\mathbf{y}^{\text{T}} B^2 \ \mathbf{y}}{\mathbf{y}^{\text{T}} \mathbf{y}}$ are called Rayleigh quotients.
My attempt:
This is not an answer but is my insufficient derivation. Let $\mathbf{a}$ replace $\mathbf{x}$ and $\mathbf{b}$ replace $\mathbf{y}$, as given below in the transformation:
$$ \begin{cases} \mathbf{a}=A\mathbf{x} \\ \mathbf{b}=B\mathbf{y} \end{cases} $$
Therefore, the simultaneous equations can be simplified using parameters $\mathbf{a}$, $\mathbf{b}$ and $\mathbf{u}\in \mathbf{R}^n$. Then, I define $\mathbf{f}$ that are vector functions.
$$ \begin{cases} \mathbf{f}(\mathbf{a}(\mathbf{x}),\mathbf{x})=\mathbf{a} - \biggl( \cfrac{\mathbf{a}^{\text{T}} \mathbf{a}}{\mathbf{x}^{\text{T}}\mathbf{x}}\biggr)\mathbf{x}=\mathbf{u} \\ \mathbf{f}(\mathbf{b}(\mathbf{y}),\mathbf{y})=\mathbf{b}-\biggl( \cfrac{\mathbf{b}^{\text{T}} \mathbf{b}}{\mathbf{y}^{\text{T}} \mathbf{y}}\biggr)\mathbf{y}=\mathbf{u} \end{cases} $$
I do not know how to calculate the next step, but I believe that this equation has a beautiful solution.
I assume that the unknowns are $x\not= 0,y\not= 0$. If $A=B=0$, then every $(x,y)$ is a solution. If $A\not= 0,B=0$, then any solution is in the form $(x,y)$ where $x$ is an eigenvector of $A$.
Otherwise $A\not= 0,B\not=0$. If $y$ is s.t. the RHS is $0$, then we return to the case $B=0$.
Now we choose $y$ s.t. the RHS is $u\not= 0$; we consider the non-linear system of $n$ equations in the $n$ unknowns $(x_i)$: $Ax-\dfrac{x^TAx}{x^Tx}x-u=0$. If $u$ is an eigenvector of $A$, then, generically, there are no solutions in $x$. If $u$ is not an eigenvector of $A$, then, by homogeneity, we may assume that $||u||=1$; generically, there are $n-1$ solutions in $x$, at least in $\mathbb{C}$; yet, in my instances, all solutions are real.