I have a (known) function $f(\rho,\phi)$ that is valid for $|\rho|>a$ (and it satifies laplace's equation)
I want to decompose it into
$$f(\rho,\phi)=\sum _{\nu\ :\ odd}C\nu J\nu(k\rho)\sin(\nu\phi)+D\nu Y\nu(k\rho)\sin(\nu\phi)$$
($\nu$ is odd because $f(\rho,\phi)=0$ for $\phi=0$ or $\phi=\pi$ and it is symmetric wrt $\frac{\pi}{2}$)
I could not find an orthogonality relation for the bessel functions that would help me. How can the coefficients be determined?
The function $1-\cos^2\phi$ vanishes for $\phi=0$ and $\phi=\pi$. Is it odd?
Otherwise $C_{\nu}J_{\nu}(k\rho)+D_{\nu}Y_{\nu}(k\rho)$ is just $\nu$th Fourier coefficient in the expansion of $f(\rho,\phi)$ with respect to $\phi$. In other words, to decompose you only need orthogonality of exponentials $e^{i\nu\phi}$, not orthogonality of Bessel functions.
Namely, given $f(\rho,\phi)$, we can write it as $$f(\rho,\phi)=\sum_{\nu\in\mathbb Z}f_{\nu}(\rho)e^{i\nu \phi},$$ where $$f_{\nu}(\rho)=\frac1{2\pi}\int_0^{2\pi}f(\rho,\phi)e^{-i\nu\phi}d\phi.\tag{1}$$ Now if $f(\rho,\phi)$ satisfies Laplace equation, then automatically $$f_{\nu}(\rho)=C_{\nu}J_{\nu}(k\rho)+D_{\nu}Y_{\nu}(k\rho).$$
The coefficients $C_{\nu}$ and $D_{\nu}$ can be identified by comparing the asymptotics of $f_{\nu}(\rho)$ defined by (1) as $\rho\to 0$ or $\rho\to \infty$ with the known behavior of Bessel functions.