Consider the following generating function of some sequence $(f_n)_n$ $$ G(f_n;x) = \sqrt{1-4a(1-a) \cdot x+(2a-1)\sqrt{1-4a(1-a) \cdot x^2}} $$ for $0\leq a\leq1/2$. How can I deduce the asymptotic formula of the coefficients corresponding to the above generating function?
The first five coefficients of the expansion around zero are: \begin{align} &x^0:\ \sqrt{2a}\\ &x^1:\ -\sqrt{2}(1-a)\sqrt{a}\\ &x^2:\ -\sqrt{2}(1-a)^2\sqrt{a}-\sqrt{2}(1-a)\sqrt{a}(2a-1)\\ &x^3:\ -3\sqrt{2}(1-a)^3\sqrt{a}-3\sqrt{2}(1-a)^2\sqrt{a}(2a-1)\\ &x^4:\ -15\sqrt{2}(1-a)^4\sqrt{a}-18\sqrt{2}(1-a)^3\sqrt{a}(2a-1)-12\sqrt{2}(1-a)^2a^{3/2}(2a-1)\\ &\ \ \ \ \ \ \ \;\ -3\sqrt{2}(1-a)^2\sqrt{a}(2a-1)^2\\ &x^5:\ -105\sqrt{2}(1 - a)^5\sqrt{a} - 150 \sqrt{2} (1 - a)^4 \sqrt{a} (2a-1) - 60 \sqrt{2} (1 - a)^3 a^{3/2} (2a-1) - 45 \sqrt{2} (1 - a)^3 \sqrt{a} (2a-1)^2 \end{align}
Life will be simpler if we simplify the OGF using trigonometry.
Since $a \in [0,\frac12]$, if we define $s = 1-2a$ and $c = \sqrt{4a(1-a)}$, we will find $s, c \in [0,1]$.
Notice $c^2 + s^2 = 1$, we can pick a $\theta_0 \in [0,\frac{\pi}{4}]$ such that $c = \cos(2\theta_0)$ and $s = \sin(2\theta_0)$.
For $0 \le cx \le c \le 1$, pick a $\theta \in [\theta_0,\frac{\pi}{2}]$ such that $cx = \cos(2\theta)$.
We can rewrite the OGF as
$$\begin{align}{\rm OGF} = \sum_{n=0}^\infty f_n x^n &= \sqrt{1 - c^2 x - s\sqrt{1 - c^2x^2}}\\ &= \sqrt{1 - c\cos(2\theta) - s\sin(2\theta)} = \sqrt{1 - \cos(2(\theta-\theta_0))}\\ &= \sqrt{2}\sin(\theta - \theta_0) = \sqrt{2}\left(\cos\theta_0\sin\theta - \sin\theta_0\cos\theta\right)\\ &= \cos\theta_0\sqrt{1-\cos(2\theta)} - \sin\theta_0\sqrt{1+\cos(2\theta)}\\ &= \cos\theta_0\sqrt{1-cx} - \sin\theta_0\sqrt{1+cx} \end{align} $$ Expand RHS by generalized binomial theorem, we find
$${\rm OGF} = \sum_{n=0}^\infty ((-1)^n \cos\theta_0 - \sin\theta_0)\binom{\frac12}{n} c^n x^\ell \quad\text{ where }\quad \binom{\frac12}{n} = \frac{1}{n!}\prod\limits_{\ell=0}^{n-1}\left(\frac12 - \ell\right)$$
Comparing coefficients of $x^n$ on both sides, we obtain a closed-form expression for the sequence $f_n$:
$$f_n = ((-1)^n\cos\theta_0 - \sin\theta_0) \binom{\frac12}{n} c^n$$
To extract the leading asymptotic behavior of $f_n$, we rewrite $\binom{\frac12}{n}$ using gamma function.
We have
$$\binom{\frac12}{n} = (-1)^n \frac{\Gamma(n-\frac12)}{\Gamma(-\frac12)\Gamma(n+1)} = \frac{(-1)^{n-1}}{2\sqrt{\pi}}\frac{\Gamma(n-\frac12)}{\Gamma(n+1)} $$
Notice for any $\lambda \in \mathbb{C}$, we have $\displaystyle\;\lim_{n\to\infty}\frac{\Gamma(n+\lambda)}{\Gamma(n)n^\lambda} = 1\;$. For large $n$, $\displaystyle\;\binom{\frac12}{n}$ behaves like $n^{-3/2}$.
As a result, the leading behavior of $f_n$ is given by following expression
$$f_n \approx -\frac{(4a(1-a))^{n/2}}{2\sqrt{\pi n^3}}\times \begin{cases} \cos\theta_0 - \sin\theta_0 = \sqrt{2a}, & n \text{ even }\\ \cos\theta_0 + \sin\theta_0 = \sqrt{2(1-a)}, & n \text{ odd } \end{cases} $$
Update
Let us switch to the side question how to extract the leading behavior of $f_n' = \sum\limits_{k=0}^n f_k$.
The OGF of $f_n'$ is given by the formula.
$${\rm OGF}' = \sum_{k=0}^n f_n' x^n = \frac{\rm OFG}{1-x} = \frac{\cos\theta_0\sqrt{1-cx} - \sin\theta_0\sqrt{1+cx}}{1-x}$$
When $a = 0$, OCF' is just constant $0$ and all $f_n' = 0$.
When $a = \frac12$, $c = 1$ and OCF' reduces to $\sqrt{1-x}$. It is easy to show $f_n'$ falls of like $n^{-1/2}$.
For the remaining cases, what happens depends on whether $a$ is far way or near $\frac12$.
When $a$ is far away from $\frac12$, everything is relatively simple.
The numerator of OCF' has two branch cuts ends at $\pm \frac1c$ and an zero at $x = 1$. This cancels out a potential pole from denominator. OGF' remains to be regular at $x = 1$ and the leading behavior of $f_n'$ is controlled by the behavior of OGF' at $\pm \frac1c$.
For $x$ sufficiently close to $\frac1c$, we have the expansion
$$\frac{\sqrt{1-cx}}{1-x} = -\frac{c}{1-c}\frac{\sqrt{1-cx}}{1 - \frac{(1-cx)}{1-c}} = -\sum_{\ell=0}^\infty\frac{c}{(1-c)^{\ell+1}} (1-cx)^{\ell+\frac12} \tag{*1a} $$
Notice for any $\alpha > 0$ and $|z| < 1$,
$$(1-z)^\alpha = \sum_{k=0}^\infty (-1)^k \binom{\alpha}{k} z^k \quad\text{ and }\quad (-1)^k\binom{\alpha}{k} = \frac{1}{\Gamma(-\alpha)k^{\alpha+1}}(1 + O(k^{-1}))\;$$
As far as leading behavior of $f_n'$ are concerned, we only need to keep the $\ell = 0$ term in $(*1a)$.
This is because contributions from the $\ell > 0$ terms have an extra factor $\frac{1}{((1-c)n)^\ell}$ attached to them. Since $a$ is far way from $\frac12$, $(1-c)$ will not be too small and we can safely ignore all of these.
Similar expansion is available when $x$ is sufficiently close to $-\frac1c$, we have
$$\frac{\sqrt{1+cx}}{1-x} = \frac{c}{1+c}\frac{\sqrt{1+cx}}{1 - \frac{1+cx}{1+c}} = \sum_{\ell=0}^\infty \frac{c}{(1+c)^{\ell+1}} (1+cx)^{\ell+\frac12} \tag{*1b} $$
Once again, contribution from the $\ell > 0$ terms have an extra factor $\frac{1}{((1+c)n)^\ell}$ attached to them. We only need to keep the $\ell = 0$ term in $(*1b)$ in the extraction of leading behavior of $f_n'$.
At the end, we find when $a$ is far from $\frac12$, the leading behavior of $f_n'$ is of the form $O(n^{-3/2})$.
$$f_n' \approx \frac{c^{n+1}}{2\sqrt{\pi n^3}}\times \begin{cases} \frac{\cos\theta_0}{1-c} + \frac{\sin\theta_0}{1+c}, & n \text{ even}\\ \frac{\cos\theta_0}{1-c} - \frac{\sin\theta_0}{1+c}, & n \text{ odd}\\ \end{cases} $$
For the final case when $a$ is close to $\frac12$, the situation is more complicated. I have no idea how to extract the leading behavior in a form depends on $a$ uniformly.
When $a \to \frac12^-$, $\frac{1}{1-c}\to \infty$. One no longer able to blindly ignore all the $\ell > 0$ terms in expansion of $(*1a)$. One consequence of this is when $a \to \frac12^-$, above approximation fails to reproduce the $O(n^{-\frac12})$ behavior of $f_n'$ at $a = \frac12$.
If one want the leading behavior of $f_n'$ for $a$ near $\frac12$, one should consider how to vary $a$ as one sends $n$ to infinity.