The widely-used form of Henon map, according to Wikipedia, is $$ \begin{cases}x_{n+1} = 1-a x_n^2 + y_n\\y_{n+1} = b x_n.\end{cases} $$
However, in some other places, for example in the manual of auto07p, there is an alternative form:
$$
\begin{cases}x'_{n+1} = a- {x'}_n^{2} -b y'_n\\y'_{n+1} = x'_n.\end{cases}
$$
My questions are: Is there a formal relation, e.g. a coordinate transformation, between these two forms of Henon map? If yes, what is the formal relation?
Yes, they are equivalent and there's a simple relation between them. It's sufficient to check a $x = \alpha u $, $y = \beta v$ change of variables. Let's do this:
\begin{cases}\overline{x} = 1-a x^2 + y\\\overline{y} = b x.\end{cases}
then:
\begin{cases} \alpha\overline{u} = 1-a \alpha^2 u^2 + \beta v\\ \beta\overline{v} = b \alpha u.\end{cases}
after that:
\begin{cases} \overline{u} = \frac{1}{\alpha}-a \alpha u^2 + \frac{\beta}{\alpha} v\\\overline{v} = \frac{b}{\beta} \alpha u.\end{cases}
And now let's choose appropriate constants:
$$ \alpha a = 1 $$ $$ b \alpha = \beta$$
At last we obtain:
\begin{cases} \overline{u} = a-u^2 + b v\\\overline{v} = u.\end{cases}
After this change of variables Henon map of form I with parameters $(a, b)$ becomes Henon map of form II with parameters $(a, -b)$.