$$\int_0^b\int_y^a \frac x{x^2+y^2}~dx~dy$$
So the region is the region below the $y=x$ line.
So my integral comes to be:
$$\int_0^b\int_0^x \frac x{x^2+y^2}~dy~dx+\int_a^b\int_x^b \frac x{x^2+y^2}~dy~dx$$
I evaluated this but I got the wrong answer (Answer is $\frac14\pi ab$) . What is the integral supposed to be
Why do you avoid integrating in the given order?
If you do this, you get one logarithmic term $\log(y^2+y^2)=\log 2+2\log y,$ which you'd want to integrate with respect to $y,$ which is easily done.
The other logarithmic term $\log (a^2+y^2)$ you may do by parts, as follows $$\int \log (a^2+y^2)\mathrm dy=\log (a^2+y^2)\int\mathrm dy -\int \frac{2y^2}{a^2+y^2}\mathrm dy,$$ which is also easily done.