Let $X$ and $Y$ be continuous random variables with joint probability distribution $f_{XY}(x,y)$.
How can I compute the probability distribution $g_X(x\mid a\le Y\le b)$?
My guess is that I have to compute the integral: $$\int\limits_a^b f_{XY}(x,y)dy$$is that correct?
Yes, you will need to evaluate that as part of the process.
$g_X(x\mid a<Y<b) ~=~\dfrac{\displaystyle\int_a^b f_{X,Y}(x,t)\,\mathrm d t}{\displaystyle\int_\Bbb R\int_a^b f_{X,Y}(s,t)\,\mathrm d t\,\mathrm d s}$