Is there a general closed form or the integral representation for the limit of the sequence: $$a_{n+1}=\frac{\sqrt{(a_n+b_n)(a_n+c_n)}}{2} \\ b_{n+1}=\frac{\sqrt{(b_n+a_n)(b_n+c_n)}}{2} \\ c_{n+1}=\frac{\sqrt{(c_n+a_n)(c_n+b_n)}}{2}$$ in terms of $a_0,b_0,c_0$?
$$L(a_0,b_0,c_0)=\lim_{n \to \infty}a_n=\lim_{n \to \infty}b_n=\lim_{n \to \infty}c_n$$
For the most simple case $a_0=b_0$ we have some interesting closed forms in terms of inverse hyperbolic or trigomonetric functions:
$$L(1,1,\sqrt{2})=\frac{1}{\ln(1+\sqrt{2})}$$
$$L(1,1,1/\sqrt{2})=\frac{2 \sqrt{2}}{\pi}$$
$$L(1,1,2)=\frac{\sqrt{3}}{\ln(2+\sqrt{3})}$$
$$L(1,1,1/2)=\frac{3 \sqrt{3}}{2 \pi }$$
$$L(1,1,3)=\frac{\sqrt{2}}{\ln(1+\sqrt{2})}$$
$$L(1,1,1/3)=\frac{2 \sqrt{2}}{3 \arcsin \left( \frac{2 \sqrt{2}}{3} \right)}$$
$$L(1,1,\sin 1)=\frac{2 \cos 1}{\pi -2}$$
$$L(1,1,\sin 1/2)=\frac{2 \cos 1/2}{\pi -1}$$
$$L(1,1,\cosh 1)=\sinh 1$$
$$L(1,1,\cosh 2)=\frac{\sinh 2}{2}$$
These values are obtained by Wolfram Alpha and Inverse Symbolic Calculator and checked with Mathematica.
This result seems familiar to me, but I'm pretty sure I don't know how to obtain a general closed form or even integral representation.
This question is closely related, but the sequence is very different.
Update:
It seems likely that the closed form for the special case $a_0=b_0=1$ is:
$$L(1,1,x)=\frac{\sinh \left(\cosh ^{-1}\left(x\right)\right)}{\cosh ^{-1}\left(x\right)}$$
However, the proof would be nice as well as the more general case.
Thanks to Sangchul Lee I now see that this sequence is exactly Carlson's transformation. Change:
$$a^2=A,\quad b^2=B,\quad c^2=C$$
$$A_{n+1}=\frac{1}{4} (A_n+\sqrt{A_nB_n}+\sqrt{B_nC_n}+\sqrt{C_nA_n})$$
$$B_{n+1}=\frac{1}{4} (B_n+\sqrt{A_nB_n}+\sqrt{B_nC_n}+\sqrt{C_nA_n})$$
$$C_{n+1}=\frac{1}{4} (C_n+\sqrt{A_nB_n}+\sqrt{B_nC_n}+\sqrt{C_nA_n})$$
Accoding to Wikipedia and references wherein, we have:
$$R_F(A_{n+1},B_{n+1},C_{n+1})=R_F(A_n,B_n,C_n)$$
$$R_F(A,B,C)=\frac{1}{2} \int_0^\infty \frac{dt}{\sqrt{(t+A)(t+B)(t+C)}}$$
Which is exactly the 'closed form' I wanted.
Here is an observation: As in your link, if $b_0 = c_0$ then we can prove recursively that $b_n = c_n$ for all $n \geq 0$. Then plugging this to OP's recurrence relation, we find that the sequence $(a_n, b_n)$ satisfies the Schwab-Borchardt recurrence relation
$$ a_{n+1} = \frac{a_n + b_n}{2}, \qquad b_{n+1} = \sqrt{a_{n+1}b_n}. $$
So if we write $(a_0, b_0) = (a, b)$, then the limit is given by the Schwab-Borchardt mean
$$ \lim_{n\to\infty} a_n = \lim_{n\to\infty} b_n = SB(a_0, b_0) = \begin{cases} \dfrac{\sqrt{a^2 - b^2}}{\operatorname{arccosh}(a/b)}, & a > b \\ \dfrac{\sqrt{b^2 - a^2}}{\operatorname{arccos}(a/b)}, & a < b \\ a, & a = b \end{cases} \tag{*} $$
Anyway, let me write down my failed attempt to mimic Carlson's proof of $\text{(*)}$ so that future me do not repeat this mistake.
Define $I(a,b,c)$ by
$$ I(a,b,c) := \lim_{R\to\infty} \int_{-R}^{R} \frac{dx}{(x+a^2)^{1/3}(x+b^2)^{1/3}(x+c^2)^{1/3}}, $$
where $z^{1/3} = \exp(\frac{1}{3}\log z)$ with the branch cut $-\pi < \arg(z) \leq \pi$. Then we can check that $I(a, b, c)$ converges. Now using the substitution $x \mapsto 4x + ab + bc + ca$, we find that
$$ I(a, b, c) = I\left( \frac{\sqrt{(a+b)(a+c)}}{2}, \frac{\sqrt{(b+c)(b+a)}}{2}, \frac{\sqrt{(c+a)(c+b)}}{2} \right). $$
This tells us that $I(a_n,b_n,c_n) = \cdots = I(a_0,b_0,c_0)$. If we recall how the AGM is computed from Landen's transformation, this may possibly allow us to analyze $L$ using $I$.
Well, it turns out that the function $I$ has a serious issue, which is that $I$ is identically zero! This can be checked either by using the fact $I(a,a,a) = 0$ or by interpreting $I(a,b,c)$ as a value of the Schwarz–Christoffel mapping.