The common density of a couple is $f(x,y)=ax+by,\text{(0<x,y<1)}$.
Which conditions should a and b satisfy?
About A: I'm almost sure the integral $\displaystyle{\iint f(x,y)dxdy}$ diverges: $$\int_0^\infty\int_{-\infty}^{1}f(x,y)dydx=\int_0^\infty axy+\frac b 2 y^2\mid_{y=1}-axy+\frac b 2 y^2\mid_{y=-\infty} dx\overset{?}{=}\infty$$. I'm almost sure I'm wrong but I don't know any other approches for solving this integral. This integral is defined iff a,b are zeroing but again I don't think that's what the question asked for.
How can I prove this integral converges?
As pointed out in the comments, $0<x, y< 1$ usually means that both $x$ and $y$ are in the open interval $(0, 1)$. With these domains, you get
$$ \begin{align} \int_0^1\int_0^1(ax+by)dxdy&=1\Leftrightarrow\int_0^1\int_0^1bydxdy+\int_0^1\int_0^1axdxdy\\ &=b\left[\frac{y^2}{2}\right]_0^1+a\left[\frac{x^2}{2}\right]_0^1\\ &=a+b\\ &=1 \end{align} $$ So $a$ and $b$ should satisfy the condition $a+b=1$.