Inverted Gamma distribution is: $$ \frac{1}{\Gamma(a)b^a} \left( \frac{1}{y} \right)^{a+1} e^{-1/by} $$ So, $$ \mathbb{E}Y = \frac{1}{\Gamma(a)b^a} \int_{0}^{\infty} \left( \frac{1}{y} \right)^{a} e^{-1/by} \ \ \ (1) $$ I think, we need to use the definition of the Gamma function: $$ \Gamma(z) = \int_0^\infty t^{z-1} e^{-t} dt \ \ \ (2) $$ In (1), how can I use Gamma function (how can I deal with $b$)?
If there isn't $b$, I think I can use $$ \Gamma(a+1) = \int_0^\infty \left( \frac{1}{y} \right)^{a} e^{-1/y} dt \ \ \ (3) $$
We have $$\mathbb{E}[Y] = \frac{1}{\Gamma(a)b^a}\int_{0}^{\infty} \left(\frac{1}{y}\right)^{a} e^{-1/by} dy$$ Now substitute $t = 1/by$, giving $dt = - dy/by^{2}$ to get
$$\mathbb{E}[Y] = \frac{1}{\Gamma(a)b^a}\int_{0}^{\infty} b\left(tb\right)^{a-2} e^{-t} dt = \frac{1}{b} \frac{\Gamma(a-1)}{\Gamma(a)} = \frac{1}{(a-1)b}$$
Note that in your comment, you did not substitute correctly.