I came across this when trying to solve the following recurrence relation via generating function. $$(4n+3)a_n+2na_{n-1}+(2n+3)a_{n+1}=0$$
If we define $A(x)=\sum_{n=0}^{\infty}a_nx^n$, we see that it satisfies the ODE $$A'+\frac{2x+1}{2x(x+1)}A=\frac{a_0}{2x(x+1)^2}$$ This equation can be solved to find two generating functions corresponding to the two independent solutions to the recurrence relation, $$A(x)=\frac{1}{\sqrt{x(1+x)}},\quad A(x)=\frac{1}{1+x}.$$
The second generating function listed tells us that $a_n=(-1)^n$ is a solution. The first generating function has a branch cut at $x=0$, it doesn't seem like there's a way to find the coefficients $a_n$ expanded near $x=0$ for this function.
Luckily, it is still possible to find the second solution: $a_n=(-1)^n\frac{(2n)!!}{(2n+1)!!}$. My question is, how are these coefficients related to the function $\frac{1}{\sqrt{x(1+x)}}$? In fact, the generating function for this solution is $$A(x)=\sum_{n=0}^{\infty}(-1)^n\frac{(2n)!!}{(2n+1)!!}x^n=\frac{\text{arcsinh}(\sqrt{x})}{\sqrt{x(1+x)}}$$
The factor $\text{arcsinh}(\sqrt{x})$ appears, and gets rid of the branch point. I am very new to the field of generating functions, this is all very confusing!
Note: I would like to understand this so I may get insight into the related recurrence relation (apologies for the alliteration)
$$(4n+3)a_n+2na_{n-1}+(2n+3)a_{n+1}=\lambda a_n$$
Where the problematic generating function is now $$A(x)=\frac{1}{\sqrt{x(1+x)}}e^{-\frac{\lambda}{2(1+x)}}$$ Here I am not so lucky as to already have the solution.
It appears that the form of the generating function you derived is incomplete. In fact, careful calculation shows that it satisfies a first order ODE similar to yours with an extra term
$$A'(x)+\frac{2x+1}{2x(x+1)}A(x)=\frac{a_0}{2x(x+1)^2}+3\frac{a_1+a_0}{2(x+1)^2}$$
Multiply by the integrating factor $\sqrt{x(1+x)}$ and rewrite to obtain
$$\sqrt{x(1+x)}A(x)=C+a_0\int dx\frac{1}{2\sqrt{x}(x+1)^{3/2}}+3(a_0+a_1)\int dx\frac{\sqrt{x}}{2(x+1)^{3/2}}$$
which evaluate to the final result
$$A(x)=\frac{C}{\sqrt{x(1+x)}}+a_0\frac{1}{1+x}+3(a_1+a_0)\left(\frac{\text{arcsinh}(\sqrt{x})}{\sqrt{x(1+x})}-\frac{1}{1+x}\right)$$
Evidently, for the choice $a_1=-\frac{2}{3}a_0$ one obtains the solution desired, so there's no contradiction.