The bigger goal is to find the antiderivative:
$$\int \frac{dx}{\sqrt{x+\sqrt{x+\sqrt{x}}}}~~~~~(*)$$
But I can settle for the definite integral in $(0,1)$. Motivation:
$$\int \frac{dx}{\sqrt{x+\sqrt{x}}}=2\sqrt{x+\sqrt{x}}-\ln (1+2\sqrt{x}+2\sqrt{x+\sqrt{x}})+C$$
This integral is easy to solve by using the following substitution:
$$x=u^4$$
$$\sqrt{x+\sqrt{x}}=u\sqrt{1+u^2}$$
Now consider the integral $(*)$. If we take $x=u^8$, we get the integral:
$$(*)=\int \frac{8u^6du}{\sqrt{u^6+\sqrt{1+u^4}}}$$
Still seems bad, and Mathematica can't solve it (or the definite integral either).
Another way I tried is by the following substitutions:
$$\sqrt{x+\sqrt{x}}=\frac{y}{2}$$
$$(*)=\int \frac{y(\sqrt{1+y^2}-1)dy}{\sqrt{1+y^2} \sqrt{2+2y+y^2-2\sqrt{1+y^2}}}$$
$$y=\sinh t$$
$$(*)=\int \frac{\sinh t(\cosh t-1)dt}{\sqrt{(\cosh t-1)^2+2\sinh t}}$$
Believe it or not, Mathematica actually solves this integral, but the resulting expression is so long and complicated, it seems useless (and by long I mean three times the size of my screen).
What do you think, is there a reasonable closed form solution for this integral? Or at least, the definite integral:
$$\int_0^1 \frac{dx}{\sqrt{x+\sqrt{x+\sqrt{x}}}}=\int_0^{\sinh^{-1} \sqrt{8}} \frac{\sinh t(\cosh t-1)dt}{\sqrt{(\cosh t-1)^2+2\sinh t}}$$
Edit:
$$\int \frac{\sinh t(\cosh t-1)dt}{\sqrt{(\cosh t-1)^2+2\sinh t}}=\sqrt{(\cosh t-1)^2+2\sinh t}-\int \frac{\cosh t~dt}{\sqrt{(\cosh t-1)^2+2\sinh t}}$$
Now let's make another substitution:
$$e^t=v$$
$$\int \frac{\cosh t~dt}{\sqrt{(\cosh t-1)^2+2\sinh t}}=\int \frac{(v^2+1)~dv}{v\sqrt{v-1}\sqrt{v^3+v^2+7v-1}}$$
Now I see the connection to elliptic integrals (which Mathematica gives as part of the answer).
We just probably need to factor:
$$v^3+v^2+7v-1$$
The limits $x \in (0,1)$ will become $v \in (1,3+\sqrt{8})$. We can also make another change of variable, leaving only one radical and getting somewhat better behaved function (finite everywhere on the real line):
$$z=\sqrt{v-1}$$
$$\int \frac{(v^2+1)~dv}{v\sqrt{v-1}\sqrt{v^3+v^2+7v-1}}=2\int \frac{(z^4+2z^2+2)~dz}{(z^2+1)\sqrt{z^6+4z^4+12z^2+8}}=$$
$$=2\int \frac{dz}{(z^2+1)\sqrt{z^6+4z^4+12z^2+8}}+2\int \frac{(z^2+1)~dz}{\sqrt{z^6+4z^4+12z^2+8}}$$