We know that
\begin{equation} X=m_0\int_{\sigma} \frac{f(P)\cos{\alpha}}{r^2} ds, \hspace{1cm} Y=m_0\int_{\sigma} \frac{f(P)\sin{\alpha}}{r^2} ds; \end{equation}
where $r$ is the length of the vector $r=\overline{P_0P}$, and $\alpha$ - the angle that this vector makes with the $x$ axis.
We want to find the attraction of an infinite homogeneous line ($f=1$) of a unit mass ($m_0=1$) located at a distance $h$ from the line.
Let us establish that the $x$ axis is a given line, and we run the $y$ axis through a point located at a distance $h$ from the line.
$$Y=\int\frac{\sin{\alpha}}{r^2} ds = -h\int\limits_{-\infty}^\infty\frac{dx}{(x^2+h^2)^{\frac{3}{2}}}$$.
How to get to the last equality? Where did the minus come from?
If $P(x,0)$ is your generic point on the $X$-axis, $A(0,h)$ is where your mass is located and $O(0,0)$ is the origin,by Pythagorean Theorem, $r^2=AP^2=x^2+h^2$. The net horizontal attraction is zero, we only need the component perpendicular to the line. So, $$ \sin\alpha=\sin (2\pi-\angle OPA)=-\sin\angle OPA=-h/AP=-\frac{h}{\sqrt{x^2+h^2}} $$ Also, $ds=dx$ (just the element of length along the line). The limits for $x$ are $-\infty$ and $+\infty$ because you need to add the contributions from each point on the line.
The "minus" sign is just indicating that the line is attracting the point, so the orientation of the total force, $Y$, is opposite to that of the $OY$-axis.