I want to expand $Y_{\mu}(a,b)$ around $b=0$. I tried to link the integral to the generalized Marcum Q-function by doing a change of variable, however I wasn't able to get a relation between the two function. Do you think that writing the modified Bessel function as an infinite series will lead to something or will it make thing more complicated? Any ideas or hints on how to tackle this problem.
Attempt:
Using the definition of the modified bessel function $$ I_{\mu}(x) = \sum_{n=0}^{+\infty}{\frac{x^{2n+\mu}}{n! 2^{2n+\mu} \Gamma(\mu+n+1)}}$$ Switching the integral and summation, I get $$ Y_{\mu}(a,b) = \frac{1}{2} \left(\frac{a}{2}\right)^{2n+\mu-\frac{1}{2}} \sum_{n=0}^{+\infty}{\frac{\Gamma(n+2\mu,b^2)}{\Gamma(\mu+n+\frac{1}{2})}}$$
where $\Gamma(a,b)$ is the upper incomplete Gamma function.
Thanks.
No need for complicated stuff like $Q-$functions:
Define ($\mu>-1/2$)
We write the integral as $$\mathcal{Y}_{\mu}(a,b)=\int_0^{\infty}f_{\mu}(x,a)dx-\int_0^bf_{\mu}(x,a)dx$$
as long as $a<1$ everything converges and the splitting is valid. Furthermore the first integral on the right is a constant independent of $b$, which we denote by $C_{\mu}(a)$.
To go further we observe that as $b\rightarrow0_+$ a taylor expansion in the second integral is valid (the arguments of any of the function contained in the integrand are much smaller then one). Therefore
$$ \int_0^bf_{\mu}(x,a)dx\sim\int_0^bx^{2\mu}\cdot(1+\mathcal{O}(x^2))\cdot \left(\frac{x^{2\mu-1}a^{\mu-1/2}}{2^{\mu-1/2}\Gamma[\mu+1/2]}+\mathcal{O}(x^{2\mu+3})\right)dx=\\b^{4\mu}\underbrace{\frac{a^{\mu-1/2}}{2^{\mu-1/2}\Gamma[\mu+1/2]4\mu}}_{D_{\mu}(a)}+\mathcal{O}(b^{4\mu+2}) $$
and
Note also that the coefficent $C_{\mu}(a)$ can be given in closed form as $C_{\mu}(a)=\frac{2^{-3/2+\mu}(1/a-a)^{-\mu} \Gamma[\mu]}{\sqrt{\pi a}}$. To prove this use term-by-term integration together with the duplication formula for the $\Gamma-$function and the series expansion for the square root.