I'm trying to determine whether or not $f$ is integrable on $E$, where $f(x,y) = e^{-xy}$ and $E = \{(x,y) : 0 < x < y < x+x^2\}$
Ok, so $f$ is continuous and non-negative on $E$ so it is measurable. This gives us that
$\displaystyle\int_E f(x,y) \ dx dy = \underbrace{\int_0^{\infty} \left(\int_{x}^{x+x^2}e^{-xy} \ dy\right)dx}_{= J} = \underbrace{\int_{x}^{x+x^2}\left(\int_0^{\infty}e^{-xy} \ dx \right) dy}_{= I} $
Then using the RHS of the last equality:
$\displaystyle\int_0^{\infty}e^{-xy} \ dx = \dfrac{1}{y}$
So,
$I = \displaystyle\int_{x}^{x+x^2} \dfrac{1}{y} \ dy = \ln(x+x^2)-\ln(x) = \ln(1+x)$
Which obviously doesn't work. What do I need to do here, do the limits on the integrals in $I$ need to be changed so that I have $0 < x < y$ and then $y$ ranging from $0$ to $\infty$?
If so, that gives:
$I = \displaystyle\int_0^{\infty} \dfrac{1-e^{-y^2}}{y} \ dy$, which I'm having trouble evaluating (but according to Wolfram, it diverges).
On the other hand, looking at $J$ I get,
$J = \displaystyle\int_{0}^{\infty}\dfrac{e^{-x^2}-e^{-x^2(1+x)}}{x} \ dx $
Which, according to Wolfram, does converge.
I'm clearly doing something silly here and I'm guessing Fubini's theorem doesn't hold at all, but I can't see why.
Thanks for any help.
First, if the range of some integral in an interated integration depends on the integration variable of an outter integral, you need to be carefull when swapping the order of integration. In your case, you have $$ E = \{(x,y) \in \mathbb{R}^2 \,:\, x,y \geq 0, x \leq y \leq x+x^2 \} \text{.} $$ That area looks like a kind of "trumpet", between the line $y=x$ and the parabola $y=x+x^2$.
For a fixed $x$, it's obvious that $y$ ranges from $x$ to $x+x^2$, and hence, as your question correctly states, $$ \int_E f = \int_0^\infty \int_x^{x+x^2}f(x,y) \,dydx \text{.} $$ This corresponds to the partitioning $E = \bigcup_{x\in [0,\infty]} \left(\{x\}\times [x,x+x^2]\right)$, i.e. a partitioning of $E$ as a disjoint set of lines parallel to the $y$-Axis. To swap the order of integration, we need to write $E$ as a disjoint union of line segments parallel to the $x$-Axis. If you fix an $y$, for $(x,y$ to lie in $E$ you must have $x \leq y$ and $x+x^2 \geq y$. The second condition is equivalent to $\sqrt{y + \frac{1}{4}} - \frac{1}{2} \leq x$, and you thus get $$ E = \bigcup_{x\in [0,\infty]} \left(\left[\sqrt{y + \frac{1}{4}} - \frac{1}{2},y\right]\times\{y\}\right) $$ and therefore $$ \int_E f = \int_0^\infty \int_{\sqrt{y + \frac{1}{4}} - \frac{1}{2}}^y f(x,y) \,dxdy \text{.} $$ Not particularly pretty, so it seems easier to use the "obvious" order of integration.
If you do that, you get $$ \int_E e^{-xy} \,d(x,y) = \int_0^\infty g(x) \,dx \text{ where } g(x) = \frac{1}{x}e^{-x^2}\left(1-e^{-x^3}\right)\text{.} $$ Let's look at $\int_0^1 g$. Then $e^{-x^2} \leq 1$ so $$ g(x) \leq \frac{1}{x}\left(1-e^{-x^3}\right) \text{.} $$ Intuitively, that RHS converges if integrated from $0$ to $1$, because $(1-e^{-x^3})$ goes to zero "fast enough" as $x \to 0$. To formalize that, use the taylor series of $e^x$ to expand $1-e^{-x^3} = x^3 - \frac{x^6}{2!} + \frac{x^9}{3!} +\ldots \leq x^3M$ for $x \in [0,1]$. This yields $g(x) \leq x^2M$ and therefore $$ \int_0^1 g \leq M \text{.} $$
Then look at $\int_1^\infty g(x)$. This is easier - you can easily find an $M$ such that $g(x) \leq Me^{-x}$ for $x \geq 1$, since you got $e^{-x^n} \leq e^{-x}$ and $x^{-1} \leq 1$ on that interval. Thus, you found a integrable dominating function, so your integral exists.
Thus, the iterated integral exists and is finite, and because $f$ is positive, so is thus the origina integral $\int_E f$.