$\int_{-\infty }^{\tau} \! x(\tau)\delta(a\tau-b\tau') \, d\tau'=\int_{-\infty }^{(a+b)\tau} \! x\left(\frac{\tau''-b}{b}\right)\delta(\tau'')\frac{1}{b} \, d\tau \mapsto 0$ for $\tau<0, \frac{1}{b}x(\frac{-a}{b}\tau )$ for $\tau>0 $
Can someone write me steps how they got the solution for case $\tau>0$ and why is upper limit $(a+b)\tau$?
I know that they used substitution $\tau''=a\tau-b\tau'$, but I am getting lost trying to get this solution and can't get this upper limit.
I think there are some mistakes here. The point of the first step is to make the argument of the delta function just be the variable of integration. So (using cleaner notation) we have
$$\int_{-\infty}^x f(y) \delta(ax-by) dy$$
and we change variables to $z=ax-by$. This turns the lower limit into $+\infty$ and the upper limit into $(a-b)x$. (I assume $b>0$.) Then $y=-\frac{z-ax}{b}$ and $\frac{dy}{dz}=-\frac{1}{b}$ So we have
$$-\frac{1}{b} \int_{+\infty}^{(a-b)x} f \left ( \frac{z-ax}{b} \right ) \delta(z) dz$$
Reversing the limits (so that the orientation is correct) changes it to:
$$\frac{1}{b} \int_{(a-b)x}^{+\infty} f \left ( \frac{z-ax}{b} \right ) \delta(z) dz.$$
Finally the integral is either $f \left ( -\frac{ax}{b} \right )$ if $(a-b)x<0$ or $0$ if $(a-b)x>0$. It is badly defined if $(a-b)x=0$.
Generally speaking, if $f$ is a continuously differentiable function which only has simple roots $r_i$, then $\delta(f(x))=\sum_i \frac{1}{|f'(r_i)|} \delta(x-r_i)$, and so $\int_a^b g(x) \delta(f(x)) dx = \sum_{i : r_i \in (a,b)} \frac{1}{|f'(r_i)|} g(r_i)$. (Again we must require that none of the $r_i$ be exactly equal to $a$ or $b$, otherwise things become badly defined again.)