Suppose we have a function $F\colon \mathbb{R}^2 \to \mathbb{R}^2$ over a numeric pair (pair of real numbers) $[a, b]$ such that:
$$F([a,b]) = [1-br+a, 1-br+a+b]$$ for some $r\in\mathbb{R}$.
find a function $G\colon \mathbb{R}^2 \to \mathbb{R}^2$ such that, for each $[a,b]\in\mathbb{R}^2$:
$$F([a,b]) = G(G([a,b]))$$
Is there a general algorithm to find such a function?
There probably is no general algorithm, but one could say that the following approach is rather general:
However, solving $B^2 = A$ might not be easy in general. In your case you could diagonalize $A$ first, $$ A = S^{-1} \begin{pmatrix}\lambda_{1} & 0 \\ 0 & \lambda_2\end{pmatrix} S, \qquad S \in \mathbb{C}^{2\times 2}, $$ but note that, depending on the sign of $r$, the eigenvalues $\lambda_{j}$ of $A$ might be complex. Then choose $$ B = S^{-1} \begin{pmatrix}\sqrt{\lambda_{1}} & 0 \\ 0 & \sqrt{\lambda_2}\end{pmatrix} S $$ which obviously solves $B^2 = A$. It does not matter which square roots you choose, but $B$ might be a complex matrix. I leave the details to you.
Remark: For $r=0$, $A$ is not diagonalizable, but it is easy to see that in this case $$ A = \begin{pmatrix}1 & 0\\ 1 & 1\end{pmatrix} = B^2 \quad\text{for}\quad B = \begin{pmatrix}1 & 0\\ 1/2 & 1\end{pmatrix}.$$