Given the recurrence relation
$$\begin{aligned} a_{n + 1} &= 2 a_n b_n\\ b_{n + 1} &= a_n^2 + b_n^2 \end{aligned}$$
with initial conditions
$$a_0 = a, \qquad b_0 = b$$
I am trying to come up with a formula for $a_n$ in terms of $a$ and $b$. I could come up with the following
$$\binom{a_{n + 1}}{b_{n + 1}} = \begin{pmatrix}b_n & a_n\\\ a_n & b_n\end{pmatrix} \binom{a_n}{b_n}$$
I'm not sure, how to proceed from here. Any ideas?
There is not going to be a linear relation as there are products on the RHS.
Note from the two recursive relations that $$ b_{n+1} + a_{n+1} = (b_n + a_n)^2\\ b_{n+1} - a_{n+1} = (b_n - a_n)^2 $$
Hence $$ b_{n} + a_{n} = (b_0 + a_0)^{2^n}\\ b_{n} - a_{n} = (b_0 - a_0)^{2^n} $$
and thus $$ b_{n} = \frac12(b_0 + a_0)^{2^n} + \frac12(b_0 - a_0)^{2^n} \\ a_{n} = \frac12(b_0 + a_0)^{2^n} - \frac12(b_0 - a_0)^{2^n} $$
is the answer.