I know that the matrix representation of the linear transformation:
$x_1=X_1+2\lambda X_2$
$x_2=X_2 - \lambda X_1$
is:
$ \left[ {\begin{array}{c} x_1\\ x_2\\ \end{array} } \right]= \left[ {\begin{array}{cc} 1 & 2\lambda \\ -\lambda & 1 \\ \end{array} } \right] \left[ {\begin{array}{cc} X_1\\ X_2\\ \end{array} } \right]$
but what if I have the non-linear transformation:
$x_1=X_1+2\lambda X_2X_1^2$
$x_2=X_2 - \lambda X_1$
How can it be expressed in matrix form to get it's inverse?
Edit:
So, if i cannot represent it in matrix form, how can I get the inverse transformation?
If $\lambda=0$, the transformation is clearly a bijection (geometrical interpretation : it is a symmetry with respect to the line with equation $y=x$),
For any nonzero value of $\lambda$, this transformation cannot be a bijection because point (0,2) would be the image of 3 different points by the inverse transformation. These points are :
$$\tag{1}(X_1,X_2)=\begin{cases}(0,2)&\ \text{it's a fixed point.}\\ (\frac{1}{\lambda}(-1+a),1+a)&\\ (\frac{1}{\lambda}(-1-a),1-a)& \end{cases} \ \ \ \text{with} \ \ \ a:=\tfrac{\sqrt{2}}{2}$$
Using other words, your system taken with $x_1=0$ and $x_2=2$, i.e.,:
$$\tag{2}\begin{cases}(i)&...&0&=&X_1+2\lambda X_2X_1^2\\ (ii)&...&2&=&X_2 - \lambda X_1\\ \end{cases}$$
has the 3 solutions given in (1).
$Remarks:$
The second and third solutions in (1) are easily found by expressing in (2)(ii) $X_2=\lambda X_1+2$ and then plugging this expression into (2)(i), getting a quadratic equation for $X_1$.
All points $(0,y)$ are fixed points of the transformation.
There are many cases where, being given $(x_1,x_2)$, there are two solutions for $(X_1,X_2)$.