I have been struggling for 6 months on finding the analytical inverse transform of a transformation below:
$$F(y,k) = 2 \int_y^{\infty}\cos\left(ka\sqrt{r^2-y^2}\right) f(r,k) \frac{r}{\sqrt{r^2-y^2}}\ \mathrm{d}r$$
where $a$ is a known constant and $\lim_{r\to\infty}f(r,k) = f'(r,k) = 0$ and all functions are continuous.
It has a similar form of Abel transform, except that it has the cosine factor in the integrand. Any help will be appreciated!
On
You have a sequence of operators $A_k$ indexed by $k$ (integers, I suppose) given by $$ (A_kg)(y) = 2 \int_y^{\infty}\cos\left(ka\sqrt{r^2-y^2}\right) g(r) \frac{r}{\sqrt{r^2-y^2}}\ \mathrm{d}r. $$ If you can invert every $A_k$, then you can find $f(r,k)$ from $F(t,k)=[A_kf(\cdot,k)](y)$ by applying $A_k^{-1}$ at each $k$.
In Abel transforms with low regularity with applications to X-ray tomography on spherically symmetric manifolds I proved with de Hoop that each $A_k^{-1}$ exists assuming you are compactly supported in $r$ — your case appears to easily satisfy our conditions. The argument works for pretty general integral equations of this kind but gives no explicit inversion formula. It does give an inversion method, though: if you divide the relevant interval in a large but finite amount of layers and use a Neumann series in each one, you get an explicit procedure. See section 2 of the linked paper for details.
Note that $F$ also depends on $a$:
$$\begin{align}F(y,k,a) &= 2 \int_0^{\infty} du \, \cos{(k a u)}\, f \left (\sqrt{u^2+y^2},k \right ) \\ &= \int_{-\infty}^{\infty} du \, e^{i k a u} \, f \left (\sqrt{u^2+y^2},k \right )\\ &= \frac{1}{k} \int_{-\infty}^{\infty} dv \, e^{i a v} \, f \left (\sqrt{k^2 v^2+y^2},k \right )\end{align}$$
because $f$ with the argument in the above integrand is even. Thus,we have a Fourier transform relationship:
$$f \left (\sqrt{k^2 v^2+y^2},k \right ) = \frac{k}{2 \pi} \int_{-\infty}^{\infty} da \, e^{-i a v} F(y,k,a)$$
or
$$f(r,k) = \frac{k}{2 \pi} \int_{-\infty}^{\infty} da \, e^{-i (a/k) \sqrt{r^2-y^2}} F(y,k,a)$$